4 Records That Make an Agentic AI Review Reproducible

A review pass that can be repeated can be tested: two runs, laid side by side.

Generative AI now does routine work in defensive document review and production, and an attorney still signs each discovery response under Rule 26(g). One question sits beneath a challenge from the other side. Could this pass be run again?

The price is low. In the Winter 2025 eDiscovery Pricing Survey from ComplexDiscovery and EDRM, GenAI-assisted review was most often priced at $0.26 to $0.50 per document, and nearly one-third of respondents reported no familiarity with or use of it. I read the two figures together: a low price can draw a firm in before anyone has asked what the tool keeps.

The technology has begun to earn its place without overturning how discovery is done, and a familiar comfort lingers beside it: a person checked the output, so the review must be defensible. Put one question to that reviewer: what was the tool told to look for? A run is made of small things that pass quickly: the instructions drafted and redrafted; the model that sat beneath the product that week; the documents pulled for each question; the chance left in every answer. In a January 2026 r/ediscovery thread on review platforms, a commenter praised a platform's AI tooling for "discrete investigative purposes" but could not imagine using it for complex matters, least of all anything "high-stakes where consistency is demanded," and consistency is made of those four small things. Lose one, and nobody can test it.

What Makes an AI Document Review Reproducible for Court?

A reproducible agentic review rests on four records: the instructions as written, the model version, the exact retrieval set and a fixed random seed, each kept in an append-only audit trail.

Reproducible means that another person, given the same materials, can run the same pass and arrive at the same coding calls. An agentic review is one where the software plans and carries out its own chain of review steps, without a person starting each one. Rule 26(g) is the federal rule under which an attorney signs a discovery response, and a challenge under it asks how that response was reached. Quinn Emanuel's Business Litigation Reports of March 19, 2026 says two things that bear on the answer: there is "no set standard" for the level of disclosure required, and the prompts that drive a generative AI review "closely resemble the instructions that would be given to a human reviewer in a document review protocol."

Check for these four before a run begins:

1. The instructions as written. The exact prompt text and the issue definitions (the written description of what makes a document responsive to each issue in the case), with every revision kept in order.
2. The model version. The exact identifier of the model that read the documents on the day of the run, which is a different thing from the name of the product.
3. The exact retrieval set. The list of documents each pass actually read, each tied to its original by a content hash, a fingerprint computed from the file that changes if the file changes.
4. A fixed random seed. The starting number that pins down the otherwise random choices a model makes as it writes, so that the same run can be asked for twice.

Think of what a run leaves behind when it is over: the tags lying on every document, thick as seed on a broad field; the rationales, full and fluent, one after another; the summary counts; and under all of it, unless someone thought to keep them, nothing of the conditions that made them. At Relevant e-Discovery our own record starts from the ground: originals are immutable and content-hashed, the audit trail is append-only, and our outputs link back to the exact exhibit they came from, so an attorney can check a call against its source before filing. The second record is the one most easily lost, a point taken up in our piece on how a review vendor can swap models mid-matter.

In 2024, an active Relativity aiR user on r/ediscovery called "the rationale, insights, and highlighting" the "hidden gems" of that product, and another commenter passed on a description of the attorney's part changing "from someone actively making the decision to someone double checking accuracy." A rationale explains a call. It does not let anyone make the call again. The common assumption is that a readable reason and a human check close the matter; they show that one answer looks right, and they leave open whether the run could be repeated.

A review of 4 sources suggests a pattern: the talk is of explaining a result and of checking it, far more than of running it again. This suggests the weak point is the run, not the reviewer. In practice, that means asking for the four records before the first document is coded.

One candid aside. Nothing I have read measures how far two runs of the same instructions drift apart, so I cannot say how often a missing seed changes a coding call; only that without the seed nobody can rule it out. Which brings the harder matter into view: why a tool that reads so fluently may still be unable to repeat itself.

Why Is It So Hard for an Agentic Review Tool to Repeat Its Own Run?

A run is hard to repeat because its result comes from tuning, and because the market divides custody records from cited answers.

Recall is the share of the truly responsive documents that a review finds. A 2024 industry comparison of review methods put generative AI recall "usually in the 70% range" straight out of the box, on a level with first-generation TAR (technology-assisted review, in which software learns from documents attorneys have already coded), and in "the 90% range" only after further testing and iteration. The difference between those two figures is not in the software. It is in the rounds.

Picture the rounds as they go: the instructions drafted, run on a sample, read against the attorneys' own calls, altered; run again, read again, altered again; slow rounds, each leaving the wording a little changed, as a path across a field is changed by every crossing, until the recall climbs and the team is content and nobody can say afterwards which wording did the work. That is the first reason the records go missing. The review a firm ends with is the last of several, and the temptation is to keep only the last.

The second reason lies under the product. Writing in 2025, a Purpose Legal author told co-panelists that the generative AI review tools then on offer were "basically the same", sharing a workflow of identifying the review set, developing prompts, validating them and applying them; the colleagues pushed back, and the author granted that the tools differ in model, prompting, interface, results and price. The same author warned: "The LLM best suited for eDiscovery review today may or may not be the best 6 months from now." So the parts that differ, and the part that changes, are the very parts a second run needs held still.

The third reason is how the market is built. Our analysis of it is plain: enterprise platforms give Bates numbering, privilege handling and chain of custody; the newer AI entrants give cited question-and-answer and chronologies; and we built Relevant e-Discovery as the combination, not one or the other. Agentic tools add a further layer. In a 2025 ACEDS article, Exterro's Kousik Chandrasekaran described agentic AI as something that "moves beyond static prompts to orchestrate tasks across identification, review, and production, while maintaining auditability and human oversight." That is a fair statement of the aim. It does not say which records leave the tool.

More steps, more places where a record must be taken. The implication is that an agentic run needs the four records at every step it orchestrates, not once for the whole matter. A cited answer shows where a fact came from, which is why I treat source-linked review as the defensibility bar; it still does not show the conditions under which the answer was produced.

Set two accounts of the same review side by side.

• Before: The AI coded the set, and our attorneys checked a sample of its calls.
• After: The final pass ran the last approved wording of the instructions, on the named model version, over the hashed document list attached, with the seed on record; every earlier wording, and the test result that retired it, is attached as well.

The first account asks to be believed. The second can be tested. And the test a small firm can run for itself, before a matter is committed to any tool, is where this goes next.

How Should a Small Firm Test an AI Review Tool's Audit Trail Before Committing a Matter?

Bring a messy collection and a hard question to the demo, run one pass, then ask the tool to show each of the four records for that run.

The records are hard to rebuild after the fact, and some cannot be rebuilt at all. A model that has since been replaced, a wording that was overwritten, a document list nobody saved: none of these comes back when opposing counsel asks. So the time to secure them is before the review starts, and the place to do it is the demo.

The demo we offer at Relevant e-Discovery stands on that footing: bring a messy collection and a hard question, and watch the tool read, code and cite your own evidence instead of reading a case study about someone else's. Bring the collection as it truly lies: the long email threads, the scanned pages gone grey, the chat exports, the duplicates heaped deep on one another; bring the question that has been troubling the case; and let the work be done in the open, slowly enough to follow. Then, of that one pass, and of any tool you are weighing, ours included, ask four questions.

1. The instructions. Show me the exact wording that just ran, and the wording it replaced. Can both be exported, with the date of each?
2. The model. Which model version answered, by its identifier and not the product's name? Will I be told if it changes while my matter is open?
3. The retrieval set. Which documents did this pass read, and which did it leave out? Can the list be exported with a content hash for each file?
4. The seed. Was the random seed fixed and written down? If the same pass is run again now, do the same calls come back?

The fourth question is the one to press. Run the pass a second time while you are still in the room. A tool that can repeat itself will show it. A tool that cannot will show that too.

Listen, as well, for the kind of answer you are given.

• A weak answer: It is all kept in the audit log, and support can pull it if you ever need it.
• A strong answer: Here is the export for the pass you just watched; the four items are in it, and the file is yours to keep.

Buyers far from litigation are asking for the same thing. In a February 2026 thread on Reddit's AMLCompliance forum, an analyst at a mid-size bank, whose team runs 15-20 active investigations monthly with 50-200 documents in each, listed a "complete audit trail of what was searched and found" among the requirements, and set aside enterprise platforms as too expensive and "overkill for our investigation volume." One reply put the rule shortly: "If an AI can't show where it found an answer, I wouldn't trust it for compliance work." Small volume does not lower the standard. It only lowers the budget for meeting it.

Two things about our own platform bear on the test. First, our processing runs single-tenant, or inside your own AWS account under your own keys, with no vendor retention and no model training; the nuance is that where the vendor retains nothing, the firm itself is the keeper of the run records, so ask where each record is stored and who can still read it after the matter closes. Second, we built for a solo or small firm on a single messy matter, the segment that enterprise platforms cannot serve without an administrator; for such a firm the records must come out of the tool as a matter of course, not through a console that someone has to be hired to run. Cost belongs in the same conversation, and our comparison of per-matter and annual platform pricing for a small firm takes up that side of the choice.

Write the four answers down, with the date and the name of whoever gave them. And if the second run comes back different from the first, note which documents moved.

What Should a Litigation Team Keep From Every AI Review Run?

Keep four records from every run: the instructions and each revision, the model version, the exact documents retrieved and the random seed, saved as the pass runs.

The courts have the subject before them now. A WilmerHale note of July 21, 2026 describes Schulte v. LinkedIn Corp., a Northern District of California decision of June 30, 2026, as addressing parties' use of generative AI tools in discovery. I cite it for its subject and its date, and I make no claim about its holding. The date is enough. I expect the argument to move, over the next year or two, from whether a tool may be used to what record of each run a party can put on the table.

A human check tests the output. It does not reach back to how the set was gathered: in 2024 a commenter on Reddit's e-discovery forum noted that each of the major review platforms was adding self-collection tools for O365, with automated workflows to process the data, so keep the collection and processing log beside the run. Left alone, a run leaves little behind it: a tag on a document; a line of reasoning; a count at the foot of a report. The slow work of saying how it all came about then falls to the attorney who signs. So start small. Pick one finished pass, ask the vendor for all four records, and note in the file which of them came back.