Evaluating eval: stop scoring models, start scoring outcomes
Benchmark accuracy is a vanity metric in production. Here's the evaluation harness we ship with every engagement.
Benchmarks measure the wrong thing for you
Public benchmarks answer a question about models in general. You have a question about one model, on your documents, for your task, at your tolerance for a particular kind of mistake. Those questions have very little to do with each other.
The gap is not subtle. A model that leads a general reasoning leaderboard can be mediocre at extracting a specific field from your supplier invoices, because your invoices have a layout quirk that appears nowhere in the training distribution and nowhere in the benchmark.
Benchmarks are useful for narrowing a shortlist from forty candidates to five. They are not useful for the decision that follows, and treating them as if they were is how teams end up defending a model choice they cannot evidence.
Building an evaluation set that is worth having
A useful evaluation set is drawn from real cases, labelled with the outcome you actually care about, and assembled before you build anything. Two hundred well-chosen cases beat twenty thousand scraped ones.
Sampling matters more than volume. A set drawn uniformly from recent traffic will be dominated by the easy majority and will barely move when the system gets better or worse at the hard cases. Deliberately over-representing the edges — the ambiguous documents, the exceptions, the ones a human had to escalate — gives you a set with discriminating power.
Label the outcome, not the intermediate. If the business result is “this claim was routed to the right queue”, label that, rather than labelling whether a particular field was extracted. Intermediate labels produce systems that optimise for the intermediate and drift away from the outcome.
Scoring what the process cares about
Accuracy as a single number hides the trade that matters. In most enterprise workflows the two error types have wildly different costs: a false positive sends a document to a human who did not need to see it, while a false negative commits a wrong value to a system of record.
So score them separately and weight them by their actual cost. Then set the operating threshold from that weighting rather than from a default. A system tuned to maximise accuracy will often sit at a threshold no operations lead would have chosen.
Where the output is generative rather than classificatory, scoring needs a different instrument — typically a rubric applied by a stronger model, calibrated against human judgements on a subset. That calibration step is not optional. An unvalidated model-as-judge is a random number generator with a confident tone.
Running evals as a gate, not a report
An evaluation that runs monthly and produces a slide is a report. An evaluation that runs on every change and blocks deployment when a score regresses is a control, and only the second one prevents anything.
That requires the eval to be fast and cheap enough to run constantly, which is another argument for a small, well-chosen set. It also requires the threshold to be agreed in advance, in writing, by someone who can say no.
The failure mode to watch for is the set going stale. Real distributions move: new document types appear, upstream systems change format, the business starts handling a case it did not before. An evaluation set that has not been added to in a year is measuring last year's problem, and it will report green while the system degrades.
What to do with the results
The score is not the deliverable. The confusion matrix is. Knowing that the system is at ninety-one percent tells you nothing actionable; knowing that nearly all remaining errors are one document type misrouted in one direction tells you exactly what to build next.
Errors should be routed back into the set with their correct outcome recorded, so that every production failure permanently raises the bar. This is the mechanism by which a system gets better over years rather than plateauing after launch.
And the results need somewhere to live that is not a notebook on someone's laptop. Versioned with the code, comparable across runs, and readable by the person who has to sign off on the deployment.
Who owns the threshold
An evaluation gate only functions if somebody has the authority to fail a release on it, and that person needs to be named before the first deployment rather than during the first regression.
In practice this is the process owner rather than the engineering lead, because the threshold encodes a business trade — how many false negatives are acceptable, and at what cost — and that trade is not an engineering decision. Engineering owns whether the number is measured correctly; operations owns what the number needs to be.
Writing that down converts an argument that would otherwise happen under deadline pressure into a decision made calmly in advance.
Common ways eval sets go wrong
Leakage is the first. If cases used to tune prompts or select few-shot examples are also in the evaluation set, the score is measuring memorisation. Split the set before you start and keep the holdout genuinely held out.
Label noise is the second, and it is more common than teams assume. Where two competent humans disagree on the correct outcome, the ceiling on any model score is set by that disagreement rate. Measuring inter-annotator agreement on a subsample tells you what your ceiling actually is, and occasionally reveals that the process definition itself is ambiguous.
Staleness is the third and the most quietly damaging, because the system reports green while the distribution moves underneath it. A standing rule that production failures are added to the set is the cheapest defence.