Same Facts, Different Answer: Legal AI’s Consistency Problem

Legal AI's Real Risk Isn't the Wrong Answer — It's a Right Answer That Silently Changes

Writing in Artificial Lawyer on 26 August 2026, Shaz Aziz of logic-engine vendor Neota Logic makes a case that deserves more attention from e-discovery buyers than it has gotten: large language models are probabilistic, so a prompt that produced a defensible answer in January can produce a different one after the vendor's next model update, with no alert and no diff to review. Aziz's fix is architectural, not procedural — keep deterministic rules engines for the decisions that must be reproducible, and use language models only for the reading and drafting work they're actually good at. The piece is unmistakably a pitch for hybrid LLM-plus-rules systems, which is exactly what Neota sells. That doesn't make the underlying question wrong: if you can't reproduce a coding decision six months from now, do you actually have a defensible record, or just a plausible one?

Why does this matter if my AI already passed accuracy testing?

Passing today's benchmark says nothing about whether the same document gets the same answer after the vendor's next model update, silently.

Most e-discovery buyers validate AI issue-coding once, at procurement, against a sample set. Aziz's point is that the validation has an expiry date nobody tracks: the model underneath a "same prompt" gets swapped by the vendor, sometimes without notice, and the prompt that governed privilege calls or responsiveness tags in January is not necessarily governing them the same way in June. For litigation, that's not a UX quibble — it's the difference between a chronology you can defend under cross-examination and one you can't reconstruct.

Does "a human reviewed it" actually protect me here?

Not by itself — presence isn't a control unless it specifies what was checked, evidences the check, and blocks the next step.

Aziz notes that "every recent 'hallucination' case that fell foul of professional standards had a human in the loop somewhere: a paralegal, a clerk, local counsel." A reviewer's name on a log doesn't tell a court what that reviewer actually verified, or whether the workflow would have proceeded regardless. For a review platform, that distinction is the whole ballgame: a control gates the workflow until a specific check is evidenced, not just witnessed.

What should I actually ask my e-discovery vendor?

Ask whether coding decisions are versioned and re-runnable, and whether model changes are logged, dated, and can be pinned.

Concretely: if opposing counsel challenges an issue-coding call made eight months ago, can the vendor show you which model version produced it, and reproduce that exact behavior on demand — or only re-run the current model and hope it agrees? Ask where the rule that made the decision actually lives: in a versioned, inspectable logic layer, or inside a prompt that changes meaning every time the underlying model is upgraded.

Where does this leave document review specifically?

Reading and summarizing documents suits a language model; the coding decision and its audit trail need a deterministic, citable record.

Aziz's framing maps cleanly onto e-discovery: an LLM can surface and summarize a document well, but the tag, the privilege call, and the chronology entry attached to it are the parts a court will later ask you to justify. If that link isn't pinned to a specific source exhibit and a specific, reproducible logic path, "the AI said so" won't survive scrutiny.

Frequently asked questions

Is this specific to one vendor's AI review tool?

No — Aziz describes a structural property of language models generally, not a defect in any single product. Any platform issuing pure LLM output without a deterministic layer underneath is exposed to it.

Does using retrieval-augmented generation (RAG) fix the problem?

RAG improves grounding by citing source documents, but citation alone doesn't guarantee the classification or coding logic itself is reproducible after a model update — that requires a separate, versioned decision layer.

Source: Artificial Lawyer.