Close Calls: An Interactive Look at the Redgrave Study: Artificial Intelligence Trends

Reviewer disagreement, not review speed, is the number to press your AI vendor on

In November 2026, eDiscovery Today's Doug Austin flagged an interactive exercise from Relativity built on the Redgrave study of its aiR for Review tool. The study compared the generative AI system with a managed first-pass team on the same 45,004 Mallinckrodt opioid-litigation documents, which are public in a University of California San Francisco archive. Relativity's write-up restates the throughput gap. aiR ran at about 45,000 documents an hour and the human team at about 770. In the time Austin spent playing, that works out to roughly 7,200 documents for the AI against 123 for the team. Players classify a handful of documents from the study and see which "team" their calls match. Austin matched the AI most often. His own takeaway was how hard classification is.

What does the throughput gap actually tell a buyer?

Only that software reads faster than people. Documents per hour says nothing about accuracy, cost or defensibility, and the vendor's own write-up supplies the figure.

The 45,000 against 770 comparison is the least contested number in the exercise, because it is a rate and not a judgment. The contested part is how well each side found responsive documents against a benchmark. Redgrave counsel's classifications set that benchmark. Relativity published the commentary. Austin notes that Gordon Cormack, Maura Grossman and Jeremy Pickens have already challenged the original findings. Treat the speed figure as a given and the quality claim as the thing to test.

Whose answer counts as right when experts disagree?

Nobody's, definitively. The Redgrave counsel, a Relativity reviewer, the AI and the review team split on many documents, so any benchmark is one lawyer's call.

By Austin's account, only a minority of the sampled documents drew agreement from every team, and a good share split evenly. If expert opinion divides that often, a headline recall figure depends heavily on whose calls were designated the gold standard. Austin's AI match is also one reviewer on a small sample that was deliberately weighted toward close calls. It illustrates the problem and proves nothing about the tool. The AI also ran on instructions written by a Redgrave partner. Prompt quality is part of the result, and your matter may not get an author like that.

What should I ask my vendor before relying on these results?

Ask who built the ground truth, how disagreements were resolved, whether you can test on your own matter, and what the tool shows to justify each call.

The coverage skips the practical point. Close calls are where responsiveness and privilege disputes actually arise. A platform that reports an average is less useful than one that surfaces the contested documents and lets a human overrule them quickly. That means each AI call should carry a rationale citing the source passage, so a second reviewer can check it in seconds. Ask for sample-based testing on your collection, with the disagreement set shown to you and not folded into a single score.

Does a public-dataset study transfer to my matter?

Partly. One opioid collection with defined issue codes is a useful test bed, but your custodians, formats and instructions will differ, so results need re-validating on your own data.

The issue codes here covered product promotion, drug samples, diversion, suspicious orders and contact with healthcare professionals. They were written for this dispute. The source does not say how the collection splits between emails and other material, so nothing here speaks to chats, scans or phone data. Run any claim through your own protocol before it reaches a court-facing workflow.

Frequently asked questions

Is the Close Calls game a validation study?

No. It is a vendor-hosted teaching exercise on a small, selected set of documents. It shows where reviewers diverge and does not measure tool performance.

Does Austin's result show AI reviews better than people?

No. He agreed with the AI most often, but he is one reviewer, and he says faster does not necessarily mean better.

Where do the original studies sit?

Austin links earlier eDiscovery Today coverage of the Redgrave study and of the Cormack, Grossman and Pickens paper that challenged it. Read both before quoting either.

Sources: eDiscovery Today, "Close Calls: An Interactive Look at the Redgrave Study" by Doug Austin, which discusses the Relativity post by Aurora Bryant.