AI Legal Research Gone Wrong: Judge Fines Lawyers for Bogus Citations

Can lawyers use AI for legal research? That’s a question many attorneys are asking as artificial intelligence tools like ChatGPT, Google Gemini, and Westlaw CoCounsel become more common in the legal profession. But a recent courtroom incident in California has turned into a cautionary tale. A federal judge issued a $31,000 sanction against two law firms for submitting a legal brief riddled with false AI-generated citations—an action that highlights the serious risks of using generative AI in legal settings without verification.

U.S. Magistrate Judge Michael Wilner didn’t hold back when condemning the involved attorneys. He criticized the use of AI tools to generate a supplemental brief containing what he called “bogus AI-generated research.” The brief cited numerous non-existent court decisions and legal quotations. These citations weren’t just inaccurate—they were entirely fabricated. Wilner stressed that “no reasonably competent attorney should outsource research and writing” to AI, making it clear that professional responsibility can't be passed off to technology.

The issue stemmed from a civil lawsuit filed against insurance giant State Farm. According to court documents, a plaintiff's attorney used Google Gemini to generate an outline for a supplemental filing. That document was then passed along to K&L Gates LLP, a global law firm, where it was integrated into the court filing without any apparent fact-checking. The result? At least two of the cited cases didn’t exist, and the final brief contained even more fabricated content—something Judge Wilner called “scary.”

This isn’t the first AI-related legal blunder to make headlines. Former Trump attorney Michael Cohen mistakenly used AI-generated case law in legal filings after misinterpreting Bard (now Google Gemini) as an advanced legal search engine. Similarly, attorneys in a Colombian airline lawsuit filed a brief with entirely fake citations sourced from ChatGPT. The legal system is increasingly encountering these pitfalls as generative AI tools become more accessible.

What makes this incident stand out is the scope of professional negligence. According to Wilner’s ruling, neither of the law firms involved bothered to validate the references before submitting them. Once the false citations were discovered, the court requested clarification, only to receive a new brief with “considerably more made-up citations.” Eventually, the responsible attorneys confessed to using both Google Gemini and Westlaw’s CoCounsel for research—AI platforms that, while powerful, still require human oversight.

Judge Wilner's decision delivers a strong message about the ethical and professional boundaries of AI usage in legal work. He emphasized that the undisclosed use of generative AI placed other legal professionals at risk and could have led to a flawed judicial ruling. “I read their brief, was persuaded (or at least intrigued) by the authorities that they cited… only to find that they didn’t exist,” he wrote. “That’s scary.”

Legal experts and scholars like Eric Goldman and Blake Reid have pointed out that the risks of AI misuse in court aren’t theoretical anymore—they’re already here. AI-generated hallucinations (fabricated outputs presented as facts) can easily mislead legal professionals if proper guardrails are not in place. As AI becomes more embedded in the legal industry, firms must implement stricter compliance protocols, including human review of AI outputs, to maintain ethical standards and avoid sanctions.

The repercussions from this case are likely to ripple across the legal field. From malpractice insurance rates to professional licensure reviews, lawyers who fail to properly vet AI-generated research may face severe consequences. Law schools, too, are under pressure to update curriculums to reflect the responsible use of AI in legal writing and research.

For firms looking to cut costs and improve workflow with legal tech, the message is clear: AI tools like Westlaw Precision, Google Gemini, and ChatGPT should be used with caution and human judgment. Legal AI is not a “set-it-and-forget-it” solution, especially when accuracy, ethics, and client outcomes are on the line.

This court ruling also fuels ongoing debates about AI regulation, transparency, and accountability across sectors. As more cases like this emerge, it’s likely that governing bodies—including state bar associations and the American Bar Association—will begin issuing formal guidelines or even sanctions against unverified use of AI in legal proceedings.

Key Takeaways for Legal Professionals:

• Always fact-check AI outputs: Regardless of how advanced the tool may seem, lawyers are responsible for the accuracy of every citation and statement submitted to a court.

• Disclose AI use: Judges may consider undisclosed AI use as misleading conduct, which can lead to ethical violations or monetary sanctions.

• Avoid over-reliance on generative AI tools: These platforms can hallucinate and should never replace human legal reasoning or due diligence.

• Adopt AI compliance protocols: Law firms should establish internal guidelines for AI usage and train their teams on potential legal pitfalls.

Artificial intelligence is transforming legal research—but not without serious risks. This courtroom misstep underscores the necessity of maintaining human oversight, ethical standards, and disclosure when using AI in the legal profession. As legal tech continues to evolve, responsible use isn’t just best practice—it’s becoming a legal obligation.