When people picture an AI courtroom disaster, they imagine something dramatic. A rogue chatbot cross-examining a witness. A judge replaced by a smug hologram. Skynet, but with a gavel.
The reality is sillier and a lot more human: a tired attorney pastes a question into a chatbot, the chatbot invents a case that doesn’t exist, and the attorney files it anyway.
It keeps happening. And every time it does, it raises the same question for anyone who cares about how technology is reshaping serious professions.
What actually goes wrong when lawyers lean too hard on generative AI?
Hallucinations Are Not a Bug, They’re a Feature
Large language models don’t know things. They predict things. Ask one for a case citation and it’ll happily produce something that looks exactly like a case citation, complete with a plausible court, a believable year, and a name that sounds like two real litigants got into a fight over a fence.
The footnotes are confident. The reasoning is tidy. The case does not exist.
This is what researchers call a hallucination, and it’s baked into how the tech works. The National Center for State Courts has a practitioners’ guide on why these tools invent citations and how to keep a human in the loop. It’s worth a read even if your only courtroom experience is watching Daredevil reruns.
The fix isn’t fancier prompts. The fix is verification. Every quote, every citation, every name gets checked against a real database before it goes anywhere near a filing.
Real Lawyers, Real Sanctions, Real Embarrassment
Stories of attorneys submitting AI-generated briefs full of fake cases have become a recurring genre. Judges are not amused. Sanctions get handed down.
Names end up in opinions that will outlive the lawyers attached to them. It’s the legal equivalent of getting a permanent achievement trophy you can never delete.
The pattern usually looks something like this:
- Tight deadline. An attorney is buried, the clock is ticking, and a chatbot promises a polished draft in seconds.
- No verification. The output reads well, so it goes into the filing without anyone pulling the cited cases to confirm they exist.
- Opposing counsel reads it. They can’t find the cases either, because the cases were never real. Now the judge knows.
- Public fallout. Sanctions, news coverage, and a cautionary tale that gets passed around every CLE seminar for the next decade.
The Tools Are Genuinely Useful, Which Is Why This Matters
Here’s the twist nerds will appreciate: the technology actually works. It’s not snake oil. A Stanford Law resource hub collects research showing that GPT-4 can deliver meaningful speed gains on legal analysis tasks when used carefully. Document review, e-discovery, first-draft memos, summarizing depositions: this is where AI earns its keep.
Industry adoption reflects that. A Thomson Reuters guide notes that roughly 26% of legal professionals now use generative AI at work, nearly double the year before. Law firms are pouring money into tech stacks the same way game studios pour money into render farms.
A separate Thomson Reuters report on the 2026 state of the US legal market found that the average law firm bumped its technology investment by 9.7%.
So the question isn’t whether to use AI. It’s how to use it without setting your reputation on fire.
What Good Legal AI Use Actually Looks Like
The firms doing this well treat AI like a junior associate with a photographic memory and zero judgment. Useful and fast, but never trusted unsupervised.
A Houston Law Review comment walks through how the ABA Model Rules on competence, confidentiality, and supervision apply when a chatbot enters the workflow. The rules didn’t change. The tools did.
- Draft, don’t decide. Use AI for first drafts, outlines, and brainstorming. Final judgment stays with a person who passed the bar.
- Verify every citation. If the model hands you a case, pull it from a real database before it touches a brief.
- Mind the confidentiality. Public chatbots are not vaults. Client facts don’t belong in a prompt window unless the tool is built for it.
- Document the workflow. If a court asks how AI was used, the answer should be specific, not improvised.
Why This Hits Harder in Criminal Defense
Plenty of legal work is high-stakes. Criminal defense is a different tier.
A hallucinated citation in a contract dispute is embarrassing. A hallucinated argument in a suppression motion can mean a person sits in a cell who shouldn’t. The asymmetry matters.
That’s why firms handling charges like DUI, weapons cases, and assault tend to be careful about which parts of the job get handed to software. A practice like this Atlantic County defense firm still leans on the unglamorous human work: pulling discovery, watching body-cam footage frame by frame, talking to witnesses who don’t return calls. AI can speed up the paperwork around that. It can’t replace it.
The robots are getting better. They’re not getting wiser. Until that changes, the lawyers who win are the ones who treat AI the way a good Dungeon Master treats a clever player: appreciate the creativity, double-check the dice.
