AI in the US legal system? California Bar sparks controversy with machine-generated questions

California Bar admits use of AI in exam. (AI Image)

The US legal education landscape is facing unprecedented scrutiny after the State Bar of California acknowledged using artificial intelligence to help draft questions for the February 2025 bar exam. The admission, made public on April 22, has sparked outrage among legal scholars and future attorneys, as concerns grow over the integrity and fairness of the exam process.
As reported by the Los Angeles Times, the State Bar confirmed that 23 of the 171 scored multiple-choice questions on the February exam were developed with AI assistance by ACS Ventures, its independent psychometrician. An additional 48 questions were recycled from a first-year law student exam, while the remaining 100 were supplied by Kaplan Exam Services under a new cost-cutting contract. The Bar has also requested the California Supreme Court to adjust the February scores, but has not acknowledged any fundamental flaws with the content itself.
Academics sound alarm over AI authorship
The revelations have drawn harsh criticism from legal educators. Mary Basick, assistant dean of academic skills at UC Irvine Law School, said, “The debacle that was the February 2025 bar exam is worse than we imagined. Having the questions drafted by non-lawyers using artificial intelligence is just unbelievable,” as quoted by the Los Angeles Times. Basick recalled that when students first raised suspicions about AI-generated questions, she initially defended the exam’s credibility—only to later discover their concerns were valid.
Katie Moran, an associate professor at the University of San Francisco School of Law, echoed Basick’s concern, stating that it was a “staggering admission” that the State Bar employed a company to draft questions using AI and then paid the same company to evaluate and approve them. Moran told the Los Angeles Times that such a process represented a clear conflict of interest, undermining trust in the validity of the exam.
State Bar defends reliability and oversight
Despite criticism, the State Bar has maintained that all questions—including those created with AI—were reviewed by legal subject matter experts and passed rigorous validation for legal accuracy and reliability. Leah Wilson, the Bar’s executive director, said in a statement to the Los Angeles Times, “We have confidence in the validity of the [multiple-choice questions] to accurately and fairly assess the legal competence of test-takers.”
According to the Bar, the full set of scored questions exceeded the psychometric reliability benchmark of 0.80. However, Moran and Basick remain skeptical, having previously flagged numerous errors in the 50 practice questions released before the exam. In a public comment reported by the Los Angeles Times, they noted that the practice items were heavily edited just weeks before the test, yet still contained “numerous errors.”
Technical failures and lawsuits deepen the crisis
In addition to the AI controversy, the February exam was plagued by logistical and technical problems. As reported by the Los Angeles Times, some candidates experienced lagging screens, disconnections, and incomplete questions during the remote administration of the test by Meazure Learning. These issues prompted a federal lawsuit against the vendor and led California Senate Judiciary Chair Thomas J. Umberg to call for an audit of the State Bar.
The California Supreme Court, which oversees the State Bar, said it was unaware of the AI’s use in question development until the April 22 announcement. A court spokesperson told the Los Angeles Times that although the court had previously encouraged the Bar to consider innovative technologies, it had not authorized their implementation in exam design.
Bar signals no return to traditional model
Despite calls from legal educators to return to the National Conference of Bar Examiners’ standardized test, the State Bar appears determined to continue with its current approach. Alex Chan, chair of the Committee of Bar Examiners, told the Los Angeles Times that nearly half of California applicants favor remote testing, and reverting to the NCBE model would eliminate that option.
Chan added that the committee would meet on May 5 to consider possible non-scoring remedies. However, he acknowledged that the State Bar is unlikely to release all 200 test questions or reinstate the NCBE-administered exams for July.
As the legal community and the public digest these revelations, the role of AI in high-stakes professional licensing remains a contentious and unresolved issue, with broad implications for the future of legal assessment in the US.