The Evaluation Gap

There is a specific kind of meeting that has repeated itself in every large organisation over the past two years. Somebody shares a screen, types a question into an agent, and the agent does something genuinely impressive. It reads three systems, reconciles a discrepancy, drafts a response, and finishes in eleven seconds. The room is convinced. A budget follows.

Six weeks later the same system is in production and the mood has changed. It is not that the agent broke. Nothing broke. It is that nobody can answer a simple question: is it working? Not is it responding, not is it fast, but is it right, and is it as right as it was last week.

The demo always works because a demo is a curated sample of one. The person driving it knows which question the system handles well, because they have asked it forty times while building. That is not dishonesty, it is proximity. Production removes the curation. Production is a thousand questions chosen by people who have never seen the architecture, against data that changed at three in the morning.

The gap between those two conditions is the evaluation gap, and it is not closed by caution, by a stricter prompt, or by a better model. It is closed by measurement. This essay is about building the instrument, because almost every piece of writing on the subject — including some of ours — argues that you should have one and then stops at the door.

The first mistake is measuring similarity instead of success. It is easy to compare the agent's output to a reference answer and score the overlap. It is also close to useless, because two responses can share almost no wording and both be correct, while two responses can be nearly identical and one of them quietly cites a definition that was retired in April.

An evaluation measures whether the task was completed correctly under the constraints that matter. For a retrieval question, that means the answer is factually right and traceable to a source that is authoritative and current. For an action, it means the correct operation ran at the correct scope, and no other operation ran. For a workflow, it means the outcome the business wanted actually happened, within budget and within time.

Notice that none of those definitions mentions the model. This is deliberate. You are not benchmarking a model, you are testing a system: instructions, retrieval scope, definitions, tool contracts, memory, and the loop that ties them together. Swapping the model is one variable in a system with fifteen. Chapter 17 makes the case for treating evaluation as core infrastructure rather than a testing afterthought, and the distinction it draws — between evaluating the model and evaluating your system — is the one most programmes get backwards in the first quarter.

There is a second, subtler point. An evaluation is a definition of correct that your organisation has agreed to write down. That is why building one is hard and why it is valuable. Half the labour is not engineering; it is getting two experts in a room to agree on what the right answer to a common question actually is. Teams that resist evals often resist that conversation, not the code.

A single end-to-end score tells you that something is wrong and nothing about where. That is the most common reason an evaluation effort dies: it produces a number that moves and no way to act on it. The remedy is to measure four layers independently, so a regression has an address.

Retrieval is the bottom layer. For a given question, did the system surface the documents, rows, or definitions that a competent human would have used? This is measurable without the model in the loop at all, which makes it the cheapest and highest-yield place to start. If the right context was never retrieved, everything above it is noise. Chapter 35 covers the mechanics — chunking, ranking, freshness against similarity, what to do when two authoritative sources disagree.

Reasoning is the second layer. Given the correct context, did the system reach the correct conclusion and stop at the right point? Failures here look like a plausible plan that goes one step too far, or an answer that ignores a constraint that was present in the context.

Tool calls are the third. Was the right tool selected, with the right arguments, at the right scope? This layer is where a system stops giving advice and starts making commitments, and it is the layer where a scoring harness pays for itself fastest, because a wrong argument to a correct tool is invisible in every other form of monitoring. Chapter 14 and Chapter 15 cover action control and the approval machinery that surrounds it.

Outcome is the top layer: did the business result occur, at acceptable cost and latency? This is the only layer an executive cares about and the only one that is useless in isolation. Measure all four or you will spend your quarter guessing.

The golden set is the asset. Everything else in an evaluation harness is plumbing that can be rewritten in a week; a well-constructed set of graded questions takes months to accumulate and cannot be bought.

Start with real questions. Pull the last few hundred requests your system actually received, or if the system is not live yet, the last few hundred tickets, emails, or queries that humans handled. Synthetic questions produced by a model are tempting and they encode the model's own assumptions about what is easy, which is precisely the bias you are trying to detect.

Then sample deliberately rather than randomly. You want the frequent cases, because they carry the volume. You want the known-hard cases, because they carry the risk. And you want the boring edge cases that experts groan about — the account with two entitlements, the region that reports on a different calendar, the product that was renamed. A hundred well-chosen items outperform two thousand scraped ones.

Each item needs four fields, and the fourth is the one everybody skips: the question, the approved answer, the sources that make it correct, and the name of the human who approved it. Without provenance and an owner, a golden set silently rots into a set of assertions nobody can defend, which is the same failure mode described in the agent org chart. Add an expiry date while you are there. A graded answer about a pricing policy is only true until the policy changes.

Finally, treat the set as a living artefact. Every production failure that a human had to correct becomes a new item that week. This is the loop that makes the harness compound: the system's mistakes become the instrument that catches the next one. Chapter 11 makes the general argument that context without review decays, and a golden set is context in the strictest sense.

There is an appealing shortcut where you hand every output to a strong model and ask it to grade the work. It is genuinely useful and it should be the last scoring mechanism you reach for, not the first, for three reasons: it costs money on every run, it drifts when the judging model changes, and it is least reliable exactly where correctness matters most, which is on domain-specific facts it has no access to.

Most of what you need to know is checkable deterministically. Did the answer cite at least one source, and is every cited source in the approved set? Does the stated figure match the number in the system of record? Did the tool call use the account identifier from the request rather than one from an earlier turn? Was the response produced within the latency and token budget? Did the agent stop, or did it hit the loop limit? None of these need a judge, all of them catch real failures, and all of them run for free in a pipeline.

Structured extraction is the middle tier. Rather than grading prose, require the agent to return the decision in fields — the recommended action, the figure, the sources, the confidence — and score the fields. This single change converts most of an evaluation problem from a language problem into a comparison problem.

Reserve the model-as-judge for the genuinely subjective residue: tone, completeness, whether an explanation would satisfy the person who asked. When you use it, pin the judging model version, publish the rubric, and calibrate it against a few dozen human-graded items so you know its bias. A judge you have not calibrated is a second unmeasured system sitting in judgement over the first.

An evaluation that runs when somebody remembers to run it is a report. An evaluation that runs automatically, and can block a change, is a control. The difference determines whether the harness survives its third month.

The rule is simple and it must be boring: any change to the prompt, the instructions, the retrieval configuration, the tool catalogue, the memory policy, or the model triggers the suite, and a drop in any layer's score beyond an agreed threshold blocks the change until a human signs off. Prompts are code. They are edited more casually than code and they alter behaviour more broadly, which is exactly the argument for putting them behind the same gate.

Two thresholds are worth defining explicitly. The first is the hard floor: a set of items that must never fail, usually anything involving money, a customer commitment, or a regulatory statement. A failure there is a blocked release, not a discussion. The second is the drift band: a few percentage points of tolerated movement on the broader set, because a suite that fails on noise gets disabled within a fortnight.

Run the suite on a schedule as well as on change, because the most dangerous regressions in an agentic system are the ones nobody caused. A source system's schema shifts, a document set is re-indexed, a definition is updated in one place and not another, and behaviour degrades while every line of your code stays identical. Chapter 16 treats observability as a first-class concern for precisely this reason.

Once the harness exists, it produces a metric that reframes the entire economic conversation: cost per correct answer. Not cost per token, not cost per request, but the total cost of arriving at an outcome your organisation would defend — including retries, escalations, and the human minutes spent verifying.

This single number resolves arguments that otherwise run for weeks. A cheaper model that is right seventy per cent of the time is usually more expensive than a costlier model that is right ninety-four per cent of the time, once you price the thirty per cent that a human has to catch. And better retrieval frequently beats both, because it improves the numerator and the denominator at once.

It also exposes the failure that no dashboard shows. A team can make an agent forty per cent cheaper and eight per cent less accurate in the same week and report only the first half, in perfect good faith, because the second half was never instrumented. The essay on small models and big context works through the arithmetic, and Chapter 18 and Chapter 19 put it in the wider frame of cost as a design constraint rather than a monthly surprise.

There is a related figure worth tracking beside it: cost of being wrong, which is not the same thing. Cost per correct answer is an efficiency measure. The cost of a wrong answer that reached a customer is a risk measure, and the two justify very different amounts of investment in the gate above.

An evaluation suite with no named owner degrades faster than the system it measures, because its decay is invisible. Nothing errors when a golden set goes stale. The suite keeps passing. It simply stops being a description of the present.

Assign four roles explicitly. Someone owns the golden set and is accountable for its freshness and provenance. Someone owns the scoring code and the thresholds. Someone owns the gate — meaning they have the authority to block a release and the obligation to unblock it quickly. And a domain expert owns the definition of correct for each area, because engineering cannot arbitrate whether a revenue figure should include intercompany transfers.

In most organisations these four collapse into two people, which is fine. What is not fine is zero. The pattern set out in the agent org chart applies directly here: every artefact gets an owner, a reviewer, an expiry, and an escalation path, and an empty owner cell is a finding rather than a formatting issue.

One organisational note. Put the harness where the engineers who change the system can see its output without asking anyone. An evaluation that lives in a weekly slide is a compliance artefact. An evaluation that appears in the same place as the build result is an engineering tool.

The reason most teams never start is that the mature version of this looks like a quarter of work. It is not. A useful first version fits in two weeks and pays for itself immediately, and every later improvement is incremental.

Days one to three: collect two hundred real requests, cluster them, and pick sixty representative items — thirty frequent, twenty known-hard, ten edge cases that experts already complain about.

Days four to six: get approved answers for the sixty, with sources and an approver name. This is the slowest step and the one to protect. Do it in a room with the domain experts rather than over email.

Days seven to eight: write the retrieval check. For each item, does the system surface at least one approved source in its top results? This alone typically finds the first real defect.

Days nine to ten: add structured output to the agent and score the fields deterministically, plus the budget checks on tokens, latency, and loop exits.

Days eleven to twelve: wire the suite into the pipeline so it runs on every change to prompts, retrieval, tools, or model, with a hard floor on the ten highest-consequence items.

Days thirteen to fourteen: publish one page showing the four layer scores and cost per correct answer, and add a standing rule that every production correction becomes a new golden item that week. That is the loop. Everything after this is refinement — a calibrated judge, per-domain reporting, a wider set — and none of it is urgent until the basics run automatically.

It is worth being clear about what an evaluation harness actually buys, because it is more than a quality gate. It buys the ability to change things.

A team without one cannot safely switch models, tighten a prompt, narrow a retrieval scope, adopt a cheaper provider, or hand a new action to an agent, because every one of those is a bet with an unobservable outcome. The same team with a harness can do all of it in an afternoon and see the effect on four numbers. That is the difference between an AI programme that compounds and one that freezes the moment it works well enough to be scary to touch.

It is also the clearest expression of the argument the whole book makes. Nobody has a durable advantage in model capability; everybody rents from the same short list of providers. The advantage sits in the context you assembled and in your ability to prove that it is working. Chapter 4 frames context as the C that gates the others, Chapter 6 explains why a definition is a decision rather than a retrieval result, and Chapter 29 argues that evaluation is the professional core of the agentic era — the skill that keeps a data career valuable while the tools underneath it churn.

The demo will always work. Build the instrument that tells you what happens on the two hundredth request, and the demo stops being the thing your programme depends on.