Reviewers Who See the AI's Tag First Rarely Overrule It

A blind QC pass: the reviewer codes the document before the AI's tag is visible.

Gen AI first-pass review gets validated on a random sample for recall and precision, but the common workflow never says whether the QC reviewer should see the AI's tag first.

The gap is the whole article. Coding blind means a reviewer reads and codes a document without seeing the AI's issue code. Coding tag-first means the AI's answer is on screen before the first line is read. Those sound like the same job. They are not. One produces an independent judgment, and the other produces a reaction to somebody else's.

Practitioners arguing about gen AI review on price make the stakes obvious. In 2025, one buyer said quotes for Rel aiR and Everlaw seemed "more expensive than offshore review," and a commenter explained the pricing model in five words: "You pay per prompt per doc." Honestly, I get the temptation that creates. When every extra look costs money, a quick confirm click on a visible tag feels like thrift (it is, right up until someone asks how you measured your error rate).

So here's my position, stated up front. A QC sample coded with the AI's tag visible measures agreement, not accuracy. Only a sample coded blind gives you an error rate that belongs to the review rather than to the AI. Whether you're weighing Relativity against other platforms or running one messy matter on something smaller, that's the number you'll need when opposing counsel asks.

Fair warning: the evidence doesn't include a measured override rate for legal review, tag-first versus blind. The argument here is about design, and it holds without one, starting with why the order of the tag matters more this year than it did before.

Why does the order of the AI tag matter so much right now?

Relevant e-Discovery's AI-assisted review runs cents per document against roughly $19K/GB for manual review, which means the human pass shrinks to QC, and QC has to be honest.

Before trusting any QC pass on AI issue codes, ask three plain questions:

1. Did the QC reviewer see the AI's code before reading the document?
2. Was the sample drawn at random, or pulled from documents the AI already flagged?
3. Is the reviewer's call logged apart from the AI's, so the two can be compared later?

Quick definitions, since the whole argument hangs on them. An issue code is the tag that sorts a document under a claim or defense in the case. A QC sample is the slice of AI-coded documents a human re-reviews to estimate how often the AI got it wrong.

A yes to the first question is the bad one. It means the QC pass measured something narrower than accuracy: how willing one person was to argue with a label already sitting on the screen. (Think of grading an exam with the answer key stapled to the front. You'd catch the howlers, at least.)

The money is why this is urgent now. At a spread like that, AI first-pass review stops being an experiment and becomes the default for any small firm that can do arithmetic, which is to say all of them. Vendors have noticed. One mass-tort vendor case study says AI tools "cut medical record review from days to minutes" and spot connections "that human reviewers often miss." Fine. Maybe. The pitch never asks what happens to the human who reads the AI's answer before reading the record.

The common assumption is that faster AI review means less human judgment is needed. I'd argue the reverse. The small amount of human judgment left in the process is now the only independent check standing between the AI's codes and your signature on a production, so it has to be clean.

On the record side, our platform keeps immutable originals with content hashing, append-only audit trails, a documented chain of custody and a fail-closed privilege gate on production, so the process holds up if opposing counsel challenges it. We also deploy inside your own AWS account, under your own keys, which settles who holds the data. There's a fuller breakdown of who controls what in a managed service versus your own AWS account if that's the part keeping you up.

None of that tells you whether the QC reviewer was looking at the answer key. An audit trail proves what happened. It cannot prove independence. Read together, these 4 sources show the same shape: cost, speed, custody and control all have answers now, and reviewer independence still doesn't.

In practice, your QC sample is where defensibility now lives. If the sample is contaminated, nothing downstream cleans it, and the next question is exactly how a visible tag does the contaminating.

Does seeing the AI's coding first bias human document reviewers?

Yes, it can. One-click links back to the exact exhibit make an AI code easy to verify, but a reviewer who starts from the tag is checking it, not coding independently.

The mechanism has a name. Anchoring is the tendency to lean on the first piece of information you see when you make a judgment. In AI-assisted review, the first thing a reviewer sees is often the AI's issue code, sitting in the coding panel before they've read a single line. The question in their head quietly changes from what is this document about? to is there any reason to disagree? Those are different questions. The second one is much easier to answer no.

Picture the same document, coded two ways:

• Tag first: The reviewer opens an email already marked responsive to a notice-of-defect issue. They skim for notice language, find a sentence that sort of fits, and confirm. Elapsed time, very little. Reasoning, borrowed.
• Blind: The reviewer opens the same email with no tag showing. They read it top to bottom, notice it's about a different product line entirely, and code it not responsive to that issue. Then the AI's call is revealed, and the disagreement goes in the log.

Same reviewer. Same document. Different answer, because the first version never asked the question that mattered. (Nobody did anything wrong in the first version, which is honestly the worst part.)

The field hasn't settled how much of this checking should happen at all. In a 2025 r/ediscovery thread on gen AI first-pass review, one practitioner said it plainly: "How much and what a team chooses to QC by humans is entirely up to them." The thread described no standard for how often reviewers check or overrule the AI's output. That silence matters. It leaves the method of QC, including whether the tag is visible, to whoever happens to set up the review.

I'd love to hand you a number here. The evidence doesn't include a measured override rate for tag-first versus blind coding in legal document review, so I won't invent one. The comparison is cheap to run on your own matter, and the result would come from your own QC logs, which is where it belongs anyway.

Here's where the tension gets real. Our outputs link back to the exact exhibit they came from, so an attorney can one-click verify before filing. That's the structural answer to the Mata v. Avianca sanctions risk, and it's why source-linked review is becoming the defensibility bar. But verifying a citation and independently coding a document are two separate jobs. Contrary to how it feels, a fast confirm click on a visible tag is not a second opinion.

We tell buyers to bring a messy collection and a hard question to the demo, then watch the platform read, code and cite their own evidence. That's the right way to decide whether to trust a tool. It's the wrong way to measure a tool's error rate on a live matter, because by then you've watched it work and you're kind of rooting for it. We also built this for the solo or small firm on a single messy matter, the segment enterprise platforms price out, and in a firm that size the person doing QC may be the same attorney who watched the AI code all week.

Small teams can't buy independence with headcount. They have to build it into the order of operations, starting with what the reviewer is allowed to see first.

How can I reduce the cost of document review in litigation and still trust the QC?

Let AI do first-pass coding on private infrastructure with no vendor retention or model training, then spend part of the savings on a small blind QC sample instead of tag-visible re-review.

Start with the sequence practitioners already use. In that same 2025 r/ediscovery discussion, one commenter laid out the gen AI workflow: test the prompts on a small set of about 500 documents, make sure the results are what you want, then "run it on a random sample for recall/precision metrics, then across the overall set (or segments of the set)." The commenter compared it to TAR 1.0, except that "you just tell AI what you want to find" instead of training the system. Another commenter there argued that the savings come from total project cost, not hourly or per-document rates. If you're coming from a TAR mindset, it's worth knowing that predictive coding isn't the only defensible AI review, and a validation sample is how the newer methods show their work.

Two terms carry the weight here. Recall is the share of truly relevant documents the review actually found. Precision is the share of documents coded relevant that really are relevant. Both numbers come from that random sample. Which is the whole problem, because if the sample is coded with the AI's tag on screen, both numbers inherit the AI's answer.

The fix doesn't add a step. It changes how one existing step is run:

1. Draw the sample at random from the full set before anyone reviews AI results for those documents.
2. Hide the AI's issue codes in the QC view, so the reviewer sees only the document.
3. Have the reviewer code each document from scratch, with a one-line reason for each call.
4. Unblind, compare the reviewer's call to the AI's, and log every disagreement before anyone resolves it.

Recall and precision then come from the blind calls, not the confirmed ones. Everything else in your QC plan can stay exactly as it is. The blind sample is the only number in the file the AI didn't help write.

On cost, I agree with the total-project framing, and it cuts both ways. Under per-prompt, per-document pricing, re-running a full tag-visible QC pass is the expensive way to learn very little. A blind random sample is a bounded slice of the set, coded once, and it produces the one error-rate figure you can stand behind.

Where the sample lives matters too. Our processing runs single-tenant, or inside your own AWS account under your own keys, with no vendor retention and no model training, so nothing your reviewers code in the blind sample trains anyone's model or waives privilege. We also keep the defensible spine (Bates, privilege, chain of custody) and the RAG side (cited Q&A and chronologies) in one tool rather than two. A QC result stored apart from the production record is one more thing somebody can ask you to reconcile.

Then there's the disagreement log, which is where this gets interesting, and also the part most QC plans quietly leave out.

What will matter most in AI review QC over the next 12 to 24 months?

Random-sample recall and precision checks will become the expected test of gen AI review, and blind-coded samples will be the ones that survive a challenge.

The middle row is the one people will argue with. The pitch for gen AI review is that it needs no training, so humans shrink out of the picture. I think the opposite happens. The training step goes away, and the human work reappears downstream as prompt testing and QC. Which is fine, honestly, as long as somebody budgets for it.

I'll be plain about the uncertainty. Nothing in the evidence measures how much a visible tag suppresses overrides in legal review. The first prediction rests on design logic and on where practitioners already are, not on a published rate.

Three developments would change the forecast:

• Courts or opposing counsel routinely accepting vendor accuracy claims without sample-based recall and precision figures.
• Per-run pricing making repeated validation samples too costly for most matters.
• Review platforms hiding AI codes by default in QC views, which would settle the question without anyone having to argue it. (I'd enjoy being wrong that way.)

Until one of those happens, the sample is yours to design, and a reviewer can't unsee a tag.

What belongs in every AI-review QC plan from here on?

A random sample coded blind, with every disagreement logged, belongs in every plan, because it's the only error-rate figure in the file that the AI's own tags didn't shape.

Here's where I think this goes. Practitioners already describe running a random sample for recall and precision before trusting gen AI across a full set. The next step is obvious, and cheap: code that sample blind. I expect blind sampling to become the checkpoint opposing counsel asks about first, because a tag-visible sample simply can't answer the question they'll ask, which is how you know the AI was right.

When first-pass review costs cents a document instead of dollars, the review stops being where the money goes. The QC sample is where the credibility goes. Even this year's eDiscovery Day is staging a live debate that puts AI on trial before a sitting federal judge. The bench is clearly curious. (So, presumably, is the other side.)

So whatever platform you land on, Relativity or one of the smaller e-discovery companies built for firms like yours, hide the tag on the sample. Log the fights. Keep the number.