Hit Counts Don't Match? Run a 5-Point Search Equivalence Test

Two hit reports can disagree without either being wrong; the settings behind them decide whether they can be compared.

Two parties can run identical search terms, swear to different counts, and both be telling the truth. Keyword search is everywhere. Its behavior is not uniform.

Miller Canfield put the matter with lawyerly dryness: virtually all review software allows keyword searching, each program with its own eccentricities in how the search is actually carried out. Magistrate Judge Paul W. Grimm said it more memorably in Victor Stanley, Inc. v. Creative Pipe, Inc. (D. Md. 2008), observing that "all keyword searches are not created equal." I believe that sentence holds an older wisdom than it seems to. So before disputing the number, ask the other side, in writing, what their engine was told to do with each term.

The eccentricities are not mysterious, though they are seldom written down. They live in five places: what a hit is, whether families are pulled in, how duplicates are removed, how text is cut into words and stemmed, and whether the index, scanned pages included, holds readable text at all. One standard search guide shows how far a single word can wander, for a query on motorcycle maintenance records returned automobile records and, in a couple of cases, email about music records. Same word. A different thing.

What follows is a way to learn which of those five settings divides two reports, before anyone spends an afternoon arguing over the terms themselves.

Which five settings decide whether two hit counts can be compared?

Two hit counts can be compared only when five settings match: what counts as a hit, family expansion, deduplication scope, tokenization and stemming, and the state of the index, including OCR.

Before any number is argued over, I would put five plain questions to each side, in this order:

1. Hit definition: Does a hit mean a document, a term occurrence or a family, and which fields were searched?
2. Family expansion: Were parent emails and attachments added to the documents that actually matched?
3. Dedup scope: Were duplicates removed across the whole collection, within each custodian, or not at all?
4. Tokenization and stemming: How does the engine cut text into words, and does "brake" also find "braking"?
5. Index and OCR state: Was every file indexed, and were scanned pages given searchable text?

The common assumption is that a list of search terms defines a search. It does not. EDRM's Search Guide, published in 2009, lists the parameters a keyword search normally carries: search string syntax, stemming on or off, wildcard syntax, case sensitivity, target data sources, field restriction and date range. For data outside English it adds character encoding, diacritics and the tokenization scheme itself. Each of those is a lever. The research section below walks through these findings.

Tokenization, I should say, is the rule by which an engine cuts a stream of text into the units it will later match. Stemming is the older and gentler practice of letting one root stand for its family of endings, so that EDRM's "brakes" reaches out to brake, braking and braked. Both are invisible in a hit report. Neither is invisible in the number.

A thread on r/ediscovery from August 2025 shows how quickly the first setting goes astray. An attorney of 20 years was told that Microsoft Purview could not return, in one search, emails matching either participant criteria or body-text criteria. The only offers were two overlapping searches, or a combined search that kept just the emails meeting both conditions. A commenter answered with a single KQL query joined by OR. The terms never changed. What a hit meant, and so what the count meant, changed entirely.

Why does this matter beyond tidiness? Every returned document is a document someone pays to read. By our own analysis of review economics, AI-assisted review runs cents per document, while manual review runs dollars per document and roughly $19K/GB, a gap the buyer can see on their own matter. Combining 3 sources points to one conclusion: an unexplained count is a budget question long before it is a legal one. Scanned records are where it hides best, as anyone who has watched OCR errors silently drop documents from a search will remember, and it is to the first two settings, the meaning of a hit and the reach of a family, that I turn next.

Is a hit count gap a real scope dispute or a settings mismatch?

Nobody can say until both counts are traced to the documents behind them. A gap that survives matched settings is a scope dispute; one that vanishes was never about the terms.

Two kinds of quarrel wear the same face across a conference table. A scope dispute is a disagreement about which documents ought to be reviewed at all. A settings mismatch is something stranger: both sides may want the same documents, yet their engines count them in different coin. Conventional wisdom reads a large count as proof of broad terms. It may instead be a count of families, or of duplicates no one removed.

Consider the same objection written twice.

• Before: Your terms return far too many documents, and we propose striking half of them.
• After: Your report counts families and ours counts only the documents that matched; once we count alike, how far apart are we?

The first opens a negotiation over words. The second opens a negotiation over arithmetic, which is shorter.

What it costs to guess wrong is not a matter of conjecture. Miller Canfield's account of In re Fannie Mae Securities Litigation, 552 F.3d 814 (D.C. Cir. 2009), tells of a third-party government agency that stipulated to a keyword process before it had seen the requesting party's list of over 400 terms. The search returned approximately 660,000 documents. The court held the agency to its stipulation, the review required 50 contract attorneys, and the agency spent over $6 million, more than 9% of its annual budget. In the firm's reading, the agency's error was surrendering control of its own methodology, so that it could never test whether the terms found responsive documents or merely documents that happened to contain them.

One might expect AI review to retire this old arithmetic. It seems, rather, to have sharpened it. Reform has disappointed before: the same firm notes that the federal e-discovery amendments effective December 1, 2006, meant to lessen the cost and risk of ESI discovery, "in many cases" had "the opposite effect." When review is priced by consumption, every document a search lets through is billed. The count is the meter.

A count, then, deserves the treatment we give an answer. In our own platform, every answer links back to the exact exhibit it came from, so an attorney can verify it in one click before filing. A hit report should be held to the same standard: a number that cannot be walked back to its documents is an assertion, not evidence. That is why the reconciliation I describe next begins on your own collection, under your own settings, long before anyone weighs reducing attorney review hours with hybrid search on a set nobody has yet measured.

How do you check equivalence on your own data before the meet-and-confer?

Rerun the terms under the other side's stated settings on your own collection, keep a record of each run, and bring five written answers. Settings travel across the table; documents stay home.

These are the steps I would carry into the room, in this order:

1. State your hit definition and ask for theirs. Is the count of documents with any hit, of unique hits per term, or of whole families, and which fields were searched?
2. Confirm family expansion. Ask for the count both with and without parents and attachments added.
3. Confirm deduplication scope. Global across the collection, within each custodian, or none at all.
4. Exchange tokenization and stemming rules. Stemming on or off, wildcard syntax, handling of noise words, and the characters the engine treats as word breaks.
5. Disclose index and OCR state. Which files failed to index, and whether scanned images were given searchable text before the run.

Each step asks for a setting, not a document. That distinction is the whole of the privilege argument.

A kitchen scale makes the point well enough. Two cooks weigh the same flour and disagree by the weight of a bowl, because one scale was zeroed with the bowl upon it and the other was not; nobody thinks to accuse the flour.

Once the other side's settings are known, I would rerun my own terms under them one setting at a time, so that each movement in the count belongs to a single cause. Change two settings at once and you learn nothing. The result is a reconciled report, and it can be laid out as plainly as this:

Where that work runs matters as much as how. In our own deployment model, processing runs single-tenant or inside the client's own AWS account under the client's own keys, with no vendor retention and no model training, so nothing fed to it trains anyone's model or waives privilege. Because originals are immutable and content-hashed, with append-only audit trails and a documented chain of custody, each rerun leaves a record meant to hold up if opposing counsel challenges it. Pairing those reruns with a privilege gate that keeps privileged ESI out of review means the check itself never exposes what it measures.

The danger on the other side is excess. The Richmond Journal of Law and Technology article on keyword search terms sets them against responsiveness, privilege and evidence standards, and names in its very title the fourth hazard, Rube Goldberg, the machine built of too many parts. One setting at a time keeps the reconciliation from becoming that machine, and the first row of the table, the meaning of a hit, is where most of these quarrels begin and, I suspect, where many of them end.

How can I reduce the cost of document review when hit counts disagree?

Settle the five settings before the terms. A gap that disappears under matched settings costs almost nothing to resolve; one negotiated blind can lock in review volume nobody measured.

I suspect the next quarrels over hit counts will turn less on which words to use and more on how each engine behaves: whether it stems, whether it reads OCR text, whether it can join metadata and body text with a single OR. The 2009 search guide I cited earlier warned that OCR text taken from scanned paper sometimes misses a letter or two, so that searches might not match. In its own example, the client's garage records surfaced only once fuzzy search was turned on. That is the whole lesson in small. The documents were there. The setting hid them.

So begin with the meaning of a hit, then families, then duplicates, then words, then the index. Bring the answers in writing. And when the other side's number still differs after all five agree, you will at last be arguing about something real.