How to Audit a Dataset Nobody Can Read

Our platform holds hundreds of millions of datapoints, and the number climbs every day. No person will ever read it. No team could sample it by hand fast enough to keep pace with what changes overnight. Is the data right this morning? Most quality assurance answers a much easier question, which is whether it was right on the day it was built.

This post sets out the pattern we landed on. It was built for supply chain risk intelligence, where the datapoints are claims about companies, commodities, and places, and it should transfer to most large evidence-derived datasets. Almost none of it is original to us.

Where this is worth doing

The pattern earns its keep on datasets built by interpretation. News coverage, NGO reporting, court filings, and audit narratives all have to be read before they become data. Judgement goes wrong in ways no schema will catch for you, because the output is structurally valid whether or not it reflects what the document said. Data that arrives with its own ground truth needs none of this, since transaction records and sensor readings can be checked against totals that either match or do not.

Take a news report about forced labour on a cocoa plantation. It names the plantation, the co-operative that buys the harvest, and an exporter that buys from the co-operative, and it quotes the exporter saying it cut ties two years ago. Is the exporter a live risk, a former risk, or a bystander? Three competent analysts can file that report three different ways, and all three versions will look perfectly valid sitting in a database row. That is the example I will keep coming back to.

Once a dataset of that kind passes a certain size, checking the contents stops being possible, because the sample you can afford to review shrinks as a proportion of the whole every month you keep ingesting. What remains checkable is the process that produces the rows. So build a small human-verified sample of what correct looks like, freeze it, and measure the pipeline against it whenever anything changes. You are no longer asking whether hundreds of millions of rows are right, which is unanswerable. You are asking whether the machine that writes them still performs to a known standard on cases where the answer is already settled.

Give the model the narrowest job that works

It is worth being clear about how little of this is done by language models, because the temptation in 2026 is to hand over a document and accept whatever structured output comes back. Most of our pipeline is deterministic and inspectable. Candidates are assembled against reference data we maintain rather than a model's impression of what counts as a commodity or a company, and a layer of rules encodes how our risk team reads a document. Those rules take precedence over anything a model proposes. Their detail is where a good deal of our work sits, so I will leave it there, beyond saying their job is to make certain mistakes structurally impossible rather than merely unlikely.

What is left for the language model is one task at the end of that chain. It confirms that a passage genuinely supports the candidate finding, and checks that the finding is attributed to the right company, commodity, and place. Attribution is where we want the judgement, because it is the call a deterministic matcher makes least reliably. In the cocoa report, the exporter might be named a dozen times because it gave a statement, while the company that owns the plantation appears once, near the bottom. It is also the call that does the most damage when it goes wrong. Keeping the remit that tight is what makes the rest of this work, because a narrow task can be measured properly and a system that both finds and judges its own findings gives you nothing stable to measure against. A 2026 review of LLM judges in healthcare found the same thing, reporting that well-scoped tasks with anchored rubrics consistently outperform open-ended generation when measured against expert judgement.

Building the standard

Everything rests on what data scientists call a gold standard corpus. A corpus is a fixed collection of documents, and gold standard means somebody trustworthy has been through it and written down the right answer for every one. Two properties matter more than size. It has to be frozen once published, so a change in the numbers means a change in the pipeline rather than a change in the test. And it has to be versioned rather than edited, so you can always see which results came from which baseline. Teams often skip this and evaluate against whatever the current system produces, on the reasonable-sounding grounds that it broadly works today. That measures consistency and calls it accuracy. A system can be perfectly consistent about being wrong, and the more consistent it is, the more confident everyone becomes.

The real objection is cost. Human labelling at any useful volume is slow, which is why most teams build a corpus once, celebrate it, and never refresh it. Two models solve most of that. Have them label the same candidate set independently, accept the label automatically wherever they agree on a straightforward case, and route everything else to a human adjudicator. Expert time then lands precisely where the machine is uncertain, and our analysts work only the cases the two models could not settle between them. That is the difference between a corpus you build once and one you can afford to keep growing. Publishing a revision still requires that every document carries a human-signed label, so the automation reduces the work without diluting who is accountable.

The evidence for letting models take the first pass is reasonably strong, though the headline numbers age quickly. In the LLM-as-a-judge literature, GPT-4 reached roughly 85% agreement with human annotators against the 81% those humans managed with each other, and that was measured in 2023, so treat it as a floor. The healthcare review above is the better guide, finding judge-expert agreement between 0.66 and 0.96 with a median of 0.83, and reporting that ensembles of models from different families cluster near the top of that range.

What the corpus has to contain

Most of a corpus is documents that should produce a result. Fill it only with those and you have tested half the job. The other half is knowing when to stay quiet, so ours also holds documents that must produce nothing at all. Testers call these hard negatives. They are traps, in the useful sense of the word.

We use three kinds, and the cocoa report supplies all of them. The exporter's denial is not evidence that the exporter did anything. A line in a risk assessment noting that child labour is common in the region is not a finding about the exporter. And the exporter appearing in one sentence of an article about a competitor's court case tells you nothing at all. All three look a great deal like a finding worth recording, which is precisely why they are in there.

These are the most valuable documents we hold, and anything that produces a claim from them has failed, whatever it scored elsewhere. Trap testing is well established. CheckList won best paper at ACL in 2020 by arguing that held-out accuracy alone overestimates real performance, then finding critical failures in extensively tested commercial systems from Microsoft, Amazon, and Google.

How you divide the corpus matters as much as what goes in it. If your evaluation set is drawn from the same documents as your training data, you are measuring memory, because a model can meet one page of a report in training and be tested on the next page, where the phrasing and house style are identical. Split at the source level instead, so whole domains go to one side of the line or the other. This single decision moves reported accuracy more than most model choices do, usually downwards, which is the point of making it. Kapoor and Narayanan's survey of leakage in ML-based science found the problem across 17 fields and at least 294 papers. In their own reproduction of civil war prediction, the supposed advantage of complex models over decades-old logistic regression vanished once the leakage was corrected.

Make the measurement binding

A measurement that produces a dashboard changes nothing. A measurement that stops a release changes everything. This is not clever on our part. Google published its ML Test Score rubric in 2017, setting out 28 tests a system should pass before going live, and gates that stop a model shipping are central to it. The idea is applied far less often than it is cited, mostly because a gate that can fail is awkward in a way a dashboard never is.

Before any change to our pipeline is allowed to write production data, it is tested against held-out documents it has never seen. Four things have to be true at once.

• Precision clears a bar set in advance. Precision is the share of the claims we produce that turn out to be correct, and the bar is agreed before the test rather than negotiated after it.
• The confidence interval clears it too. A small sample can produce a flattering average by luck, so the bottom of the likely range has to clear the bar as well as the headline figure.
• Enough cases were tested. A thin slice of documents cannot carry a release.
• Not one trap case produced a claim.

Clear all four and production writes are switched on. Fall short on any one and they stay off. The previous version stands until something better passes, and there is no way to skip the test. That last part is what makes the result mean anything months later, when nobody remembers the release.

The clearest example came from a routine model upgrade. Every headline number improved, and at a glance it was plainly better than what it replaced. It failed on one condition. Trap cases had started producing claims, because the new model was reading denials as confirmation. Our cocoa exporter would have been flagged on the strength of its own statement saying it had cut ties.

No aggregate number would have shown that. The affected documents were a small share of the test set, easily absorbed into a precision figure that still looked strong. They were not random, though, and shipping the change would have meant recording claims against companies on the strength of those companies denying them. The traps caught it in an afternoon and the upgrade never went live.

Keeping the standard alive

Human adjudication is the expensive part and the place the expertise sits, so capture that expertise as it is spent. Every adjudicated case, along with the reasoning behind it, feeds written guidelines precise enough for an AI adjudicator to apply consistently. Over time the adjudication itself becomes AI-driven, and expert time moves up a level to arbitrating the guidelines and the hardest cases the system surfaces. The gate does not move while that happens. However the labelling gets done, the pipeline still proves itself against a frozen human-signed standard before it touches anything a client sees.

Plan for the standard going stale, too. Publications change how they name companies, new sources and document types appear, and the mix of documents flowing through this quarter drifts away from the mix sampled when the corpus was built. Watch what is arriving rather than only the accuracy figures coming out, because a shift in sources or formats means your measurement is drifting out of scope even while the numbers hold steady. Then version the standard properly, so that when performance drops you have a clean question to ask. Did the pipeline get worse, or did the baseline change? Teams that edit their gold set in place can never answer that, and end up debugging phantom regressions.

What the pattern does not solve

None of this rescues a poor evidence base, since a perfectly accurate extraction from a bad source is still a bad datapoint. A gate is also a floor rather than a guarantee. It tells you a system met a defined standard on a defined corpus on a particular day, and that corpus is only ever as good as the people who built it. Nor does any of it make the underlying judgements easy. What it does is make them visible, repeatable, and expensive to get wrong quietly.

The pattern comes down to a fixed standard and an honest split, then a narrow role for the model sitting behind a hard gate that stops bad work reaching production. What makes it worth the effort is that it holds at hundreds of millions of rows as well as it does at a thousand, and it keeps holding while the dataset grows underneath you.