Building Your Own AI Review Agent: Where It Breaks

Wiring the workflow is the easy part. Proving what it missed is not.

A single engineer can build a working AI review agent from a webhook, a container job and a command-line model, and even its builder calls the result a love-hate relationship.

According to Karl Stoney, who wrote up his pull-request reviewer for Autotrader's platform in October 2025, "one minute I see the future, the next minute I git revert everything it's done and just write it myself." That is the honest version of every tutorial.

Here is the situation for a small litigation firm. The collection is large, the budget is small, and the tutorials make a homemade review agent look like an afternoon project. The complication is that software teams have already run this experiment. One vendor that benchmarks review tools reports its production reviewer fixed 1.03 bugs per pull request against 0.54 for human reviewers, and the same vendor argues that quality cannot improve without measurement. Teams without evaluations, another engineering group warns, end up guessing and checking.

So the question is not whether you can build one. You can. The question is whether you can defend what it did.

My answer is that a demo pipeline skips three things: a recall measure, a privilege gate and an immutable audit trail. Build-your-own trades a weekend of setup for an un-defendable record. In this article, I will walk through the demo pattern, each of the three gaps, a defensibility checklist and a build-or-buy test.

What does a do-it-yourself AI review pipeline actually include?

Across our clients' day-to-day work, AI-assisted review runs cents per document versus dollars per document, and roughly $19K/GB for manual review. That gap pulls small firms toward building.

An analysis of 3 sources shows the same skeleton in every public build. Karl Stoney's pull-request reviewer at Autotrader, Pasi Huuhka's reviewer on Azure, and a widely shared CI tutorial all connect an event to a model and post whatever comes back. Swap the pull request for a custodian upload and the comment for a coding decision, and you have a document review agent.

I call the shared design the four-part demo pattern. Each part is ordinary software:

1. Trigger. An event starts the run. According to Stoney, a GitHub Enterprise webhook on a pull-request event hits a code-review API, which checks a per-repository config file before doing anything else.
2. Isolated job. Each review runs in its own container. Stoney schedules a Kubernetes Job; Huuhka starts one Azure Container Apps job per review.
3. Model call. The job hands the material and a rubric to a model. The tutorial build lets you pick OpenAI, Anthropic, OpenRouter or a local Ollama model by swapping environment variables.
4. Posted output. The result lands where people will see it, usually as a comment on the pull request.

Translated to a litigation matter, the same pattern looks like this: a new production volume lands in a shared folder, a job picks up each document, a model tags it responsive or not with a short reason, and the tag appears in a spreadsheet. Nothing about that design is exotic, which is exactly why it spreads.

The reality is that the model is the smallest part of the system. Huuhka writes that "the AI part is surprisingly small," and he describes a review bot as "normal event-driven software with an LLM in one stage." Stoney locates the difficulty elsewhere: "The meat of the complexity is in the prompt." His prompt even carries an explicit list of things not to flag, built from false positives in earlier reviews.

The economics make the pattern hard to resist. The tutorial author argues that a homemade reviewer beats a subscription because it runs only when work arrives, bills by use, and can route to a local model where the marginal cost is near zero. For a solo litigator asking how to cut the cost of document review, that argument sounds like the whole answer.

In practice, the plumbing is not where these projects struggle. The difficulty sits in the instructions and in everything wrapped around them.

Look closely at what the four parts are built to do, though. Every one of them moves material forward: fire, run, read, post. None of them looks backward to ask what the model skipped. None of them stops a sensitive item from leaving, and none keeps a record that cannot be changed later. The takeaway is that these builds were designed to produce output, not to prove it.

In summary, a do-it-yourself review agent is four pieces of routine software wrapped around one model call, and the per-document economics make it tempting. The pattern answers whether a model can review your documents. It does not answer whether you can show how the review was done.

How do you know an AI reviewer isn't missing responsive documents?

You know only by measuring recall against a verified sample of your own documents. Most do-it-yourself builds never create that sample, so nobody can say what was missed.

Three terms carry this section. Recall is the share of truly relevant documents the review actually found. Precision is the share of flagged documents that turn out to be relevant. Elusion is the rate of relevant documents still sitting in the pile the review set aside. A demo reports none of them.

Why do test scores overstate real performance?

Clean test data flatters every agent. One practitioner's email-triage agent tested at 95% on clean data and 68% on real emails. The failure breakdown: 12% typos breaking keyword matching, 8% mixed-language emails, 7% sarcasm or tone detection, and 5% complex multi-part questions. A later shadow run found that forwarded email chains broke the agent and HTML formatting broke its responses.

Read that list as a litigator. Forwarded chains, stray HTML and half-finished thoughts are the normal texture of a custodian's mailbox. In practice, a demo score tells you how the agent handles the demo.

How much does recall vary between tools?

Widely. According to Augment Code, which published benchmark results for AI code review tools, its own reviewer reached 62.8% recall against 53.3% for Copilot. It is a vendor grading its own category, so read the ranking with care. The spread is the point. Graphite posted the highest precision in the comparison, 75.0%, yet "found only 12 of 137 bugs (8.8% recall)." That is a tool that is rarely wrong and usually absent.

The more useful part of Augment's write-up is the method. It built a ground truth of "golden comments" from two sources: bugs human reviewers had found, and valid issues that agents found but humans missed, each checked by hand. Every tool's output was then scored against that set. The document-review equivalent is an attorney-coded sample that the agent's calls are measured against. That sample is exactly the recall and elusion validation a demo pipeline omits.

What does grading the outcome mean?

According to Anthropic's engineering team, an agent's transcript and its outcome are different things. A flight-booking agent "might say "Your flight has been booked"", but the outcome is whether a reservation exists. A review agent that reports no responsive documents in a batch is making the same kind of claim. Anthropic also describes how Descript moved from manual grading to LLM graders with periodic human calibration, a workable model for a firm that cannot hand-check everything.

Noise is the other half of the problem. An engineer at a large tech company estimated that about 80% of their employer's in-house AI review comments were "lacking context, blatantly incorrect, or insignificant nitpicks." High noise trains reviewers to skim. Skimming is how misses go unnoticed.

Before and after. Here is the same validation claim, written twice:

• Before (demo): The agent reviewed the collection and flagged the responsive documents.
• After (defensible): An attorney coded a random sample drawn from the documents the agent set aside; the relevant documents found in that sample estimate what the agent missed, and the sample, reviewer and result are on file.

The takeaway is that recall must be measured on your collection. Firms that already run technology-assisted review will recognize this question. A homemade agent inherits the question without inheriting any answer.

In summary, I would not rely on any AI review, built or bought, that cannot show a recall figure from a sample of the matter's own documents.

What will matter most for AI review agents in the next 12-24 months?

Proof will matter more than speed. Review agents will be judged on whether they can show what they missed, keep a record of each run, and stop before acting.

I see three signals behind that forecast. None comes from a courtroom yet. All come from software teams that hit the same wall first, and each maps cleanly onto a document review queue.

The uncertainty is real, and I would rather name it than hide it. A widely accepted recall benchmark built into commercial review tools would make homemade validation layers less necessary. A slowdown in review volume would ease the pressure to triage. Either would weaken this forecast.

What most buyers miss: the loudest bet is on full autonomy, and for high-stakes review the evidence points the other way. According to Bryan Finster, only two categories still justify a blocking human gate: tribal knowledge, and regulated paths where separation of duties is a compliance requirement. Privilege review sits in both. Finster also cites SmartBear's analysis of a Cisco Systems team, which found defect detection "degrades sharply" past 400 lines, and he puts the lesson bluntly: "You cannot hire your way out of a sampling rate problem." His answer is automation plus a narrow, deliberate human gate. That is a design. A demo is not.

What keeps privileged documents protected and the review provable if challenged?

Two controls do that work: a fail-closed privilege gate on production, and append-only audit trails over immutable, hashed originals. In our experience, that combination holds up under challenge.

Two definitions first. A fail-closed gate blocks an item whenever it is unsure or broken, instead of letting the item through. An append-only audit trail accepts new entries but never allows an old entry to be edited or deleted. A demo pipeline has neither.

Where does privileged material leak in a do-it-yourself build?

It leaks at the model call. According to Karan Patel of Redfox Cybersecurity, whose firm sells secure code review services, LLM API endpoints are an attack surface for "prompt injection, data exfiltration." His example is blunt: "If a developer accidentally commits an AWS access key and your agent sends that file verbatim to an OpenAI endpoint, you have just exfiltrated credentials to a third-party API."

Replace the access key with a privileged memo and the sentence still works. A homemade review agent sends each document, in full, to whatever endpoint sits in its environment variables. Patel adds that most implementations "expose review output in unprotected logs." The recommended defense against instructions hidden inside the reviewed material is a blocklist of 8 regex patterns, such as "ignore previous instructions." That is a thin wall to put in front of a privilege log.

What this means is simple. Every document sent to a hosted model is a disclosure decision.

What does a working privilege gate need to do?

I would hold any gate to three requirements:

1. Sit between review and production. Nothing reaches a production set without passing it.
2. Fail closed. Ambiguous, low-confidence or errored items are held, not released.
3. Record the release. A named attorney clears each held item, and the clearance is logged.

Where the model runs matters as much as the gate. In our deployments, processing runs single-tenant or inside the client's own AWS account under the client's own keys, with no vendor retention and no model training, so nothing fed into it trains anyone's model or waives privilege. That posture, the firm's own cloud and the firm's own keys, is one no consumer AI tool and almost no small-firm tool can match.

What makes the review provable later?

Three records turn a review into evidence:

• Immutable originals with content hashing. A hash is a digital fingerprint of a file; change one byte and the fingerprint changes, so any alteration shows.
• An append-only audit trail. Every coding call, gate hold and release is written once and never rewritten.
• Documented chain of custody. Each original's handling history is recorded from collection to production.

Compare that with the demo: tags in a spreadsheet anyone can edit, and logs nobody protected. The takeaway is that a log you can edit is a diary, not evidence.

In summary, a defensible review needs a gate that refuses by default and a record that cannot be rewritten. A demo pipeline supplies neither, so a firm that builds one has to add both before the first production leaves the office.

Where does that leave a firm tempted to build its own?

Build to learn, not to produce. A homemade agent is a fine teacher, but a live matter needs recall evidence, a gate and a permanent record before anything leaves.

The evidence points one way. An agent that looked nearly perfect on clean test data lost 27 points on real email, and a demo never finds that out. Benchmarks from the code world show recall swinging widely between tools, and the teams that ship dependable agents grade outcomes, not transcripts. I expect review agents to be judged less on speed and more on whether they can show what they missed, keep a record of every run, and stop before acting on an uncertain call. One major model provider's agent-building guide already describes an agent that halts and hands control back to the user. That is the posture document review needs.

Karl Stoney put the right relationship well. With AI review in place, he says, "I truly own the code I put into production, but I have an assistant looking for my inevitable mistakes." Swap code for a production set and that is the standard. The attorney owns the call. The assistant catches mistakes; it does not make the final one.

In my view, the practical next step is modest. Before you wire anything, write down what your rubric requires the agent to report: what it reviewed, what it assumed, and what it did not check. Then ask who holds the sample that proves the rest.