AI Penning California’s Bar Exam: A Recipe for Chaos

The State Bar of California has acknowledged that 23 questions from the February 2025 bar exam were generated with the assistance of artificial intelligence, rather than by licensed attorneys.

The bar now plans to petition the California Supreme Court to modify scores due to issues with the test, which also experienced significant technical difficulties.

This licensing exam, crucial for aspiring lawyers, faced numerous challenges, including AI-generated questions, system failures, and lawsuits from examinees.

• 171 total scored multiple-choice questions
• 100 from Kaplan
• 48 derived from an older first-year exam
• 23 were created with AI assistance by ACS Ventures
• $8.25M: Kaplan’s contract value
• $22M: Budget deficit faced by the State Bar

Reactions to the situation

Legal educators are deeply concerned.

“It’s unbelievable that these questions were formulated by non-lawyers using AI,” remarked Mary Basick, assistant dean at UC Irvine Law.

“This is a shocking revelation,” added Katie Moran, a law professor at USF. “The same company that utilized AI also sanctioned its own questions.”

Test-takers in February reported issues such as:

• Platform crashes
• Problems saving essays
• Errors in copy-paste functionality
• Questions that were nonsensical or riddled with typos

Meazure Learning, the provider of the testing platform, is now facing lawsuits from affected students.

Looking ahead

AI is already making waves in the legal industry, but a lack of oversight and efforts to cut costs have turned this initiative into a debacle. Dean Andrew Perlman of Suffolk Law indicates that while AI could aid in exam preparation, it must be monitored closely by legal experts.

“In the future, our focus may shift to the competency of lawyers who do not leverage AI,” Perlman projected.

Upcoming developments:

• Supreme Court consideration of score adjustments

<