Small Models, Big Context

There is a moment that arrives in every agentic programme somewhere between the pilot and the second quarter of production. The invoice lands, somebody opens it expecting a number they can defend, and the number is roughly four times what the model pricing page implied. The team does the honest thing and starts looking for waste. Somebody suggests moving the workload to a smaller, cheaper model. It sounds obviously correct: the same task, a fraction of the per-token price, an easy win to present upstairs.

A month later the invoice is lower and the total cost of the programme is higher. Support tickets have gone up. A reviewer has been added to the loop for a class of answers that used to pass unattended. The agent now takes three attempts at tasks it used to complete in one, and each attempt drags a full context window behind it. The saving was real at the token level and imaginary at the outcome level.

This is not an argument against small models. It is an argument against measuring the wrong thing. The price of a token is the most visible number in an agentic system and one of the least important. What actually determines cost is how often the system is right the first time, and that is governed far more by the quality of the context it receives than by the size of the model reading it.

The unit that matters is cost per correct answer, delivered at the confidence level the business requires. Everything else is an input to it. A model that costs a tenth as much per token but requires three attempts, a fallback to a larger model on failure, and a human reviewer to certify the result is not a tenth of the cost. It is frequently a multiple of it, and the multiple is hidden in payroll rather than in the platform bill.

Writing the unit out makes the structure obvious. Cost per correct answer equals the cost of one attempt, multiplied by the expected number of attempts, plus the cost of any escalation, plus the cost of the review that determines whether the answer is acceptable, plus the expected cost of the errors that survive review. The last term is the one nobody models and the one that dominates in regulated or customer-facing work.

This is the framing the book uses in Chapter 18, which treats cost as an architectural property rather than a procurement outcome, and it is why Chapter 19 insists on unit economics that a finance function can actually audit. If you cannot state the cost of one correct answer for your three most common agent tasks, you do not have a cost problem yet. You have a measurement problem, and it will become a cost problem on somebody else's timeline.

Instrument a real agentic workflow and the distribution is consistently surprising. Retrieval is cheap. Tool calls are moderate and predictable. Generation tokens are meaningful but rarely dominant. Retries are enormous, because a retry pays for the entire context window a second and third time and usually adds reasoning on top of it. Human review is larger still, because it is priced in salaried minutes rather than fractions of a cent.

There is a fifth line that almost never appears anywhere: the cost of an error that escapes. A wrong number in a board pack, a wrong entitlement quoted to a customer, a wrong metric that gets copied into a downstream report and becomes the basis for a decision. The book treats this directly in the essay-length argument of the cost chapters, and the wider consequence is described in the pillar page on cost. An error that escapes is not a one-off expense. It is a liability with a long tail.

Once the distribution is visible, the optimisation strategy inverts. Reducing the price of an attempt saves single-digit percentages. Reducing the number of attempts saves multiples. Everything that improves first-pass correctness is therefore a cost lever, even when it looks like a quality project on the roadmap.

A larger model is better at reasoning over what it has been given. It is not better at knowing what it was never told. When an agent gets an enterprise question wrong, the failure is very rarely a reasoning failure. It is a knowledge failure: the definition of the term was ambiguous, the retrieved documents contradicted each other, the freshest source lost to the most similar one, the tool description promised more than the tool delivered.

None of those failures are repaired by parameters. They are repaired by definitions, retrieval design, tool contracts, and memory — the four surfaces the book calls the context layer, argued in Chapter 4 and given its mechanical treatment in Chapter 35. Chapter 6 makes the sharper version of the claim: meaning is a decision the organisation makes, not a pattern a model discovers, and no amount of scale converts one into the other.

This is why the cost curve of context and the cost curve of scale behave so differently over time. Buying a bigger model gives you a step change once and then a flat line. Fixing a definition, tightening a retrieval scope, or correcting a memory entry reduces the failure rate of every future task that touches it. Context improvements compound. Model upgrades do not.

The teams that have genuinely reduced agent cost did not standardise on one model. They built a router, and they routed on the shape of the task rather than on the seniority of the requester. Most enterprise work is narrow: classify this, extract those fields, summarise this thread, draft a reply from this template, look up an entitlement and state it plainly. Narrow work with well-specified context is exactly where a small model is not merely adequate but preferable, because it is faster and its failure modes are easier to characterise.

The middle tier handles work that requires synthesis across several sources or a judgement call that has more than one defensible answer. The frontier tier handles genuinely open-ended work, adversarial inputs, and anything where the cost of being wrong is high enough that paying more for one attempt is obviously correct.

The important part is the escalation rule, and it must be explicit rather than emergent. Escalate when retrieval confidence falls below a threshold, when sources disagree, when the task touches an entity the system has no definition for, or when the requested action crosses a permission boundary. That last condition is a control decision as much as a cost decision, which is why it belongs with the material in Chapter 14 and Chapter 15 on action control and guardrails.

An agent that takes ninety seconds to answer is not merely slower than one that takes six. It is a different product. People stop waiting, switch to the old process, and the workflow you funded quietly becomes a demonstration rather than a habit. The compute bill goes down for a reason that should alarm anybody who reads it carefully: nobody is using the system.

Latency also compounds inside multi-step workflows. Five sequential steps at eight seconds each is a forty-second answer, and the tail is much worse than the average because one slow retrieval or one retry doubles the whole chain. This is the argument made at length in the essay on the compute and latency budget, and it is the practical reason small models earn their place: not because they are cheap, but because they are fast enough to keep the workflow alive.

The book connects this to adoption rather than to infrastructure. Chapter 27 argues that the systems which change how an organisation works are the ones that are fast enough to be used without thinking, and that a correct answer delivered after the decision has been made has a value of zero.

Long-running agents accumulate context. Conversation history, tool outputs, retrieved documents, intermediate reasoning. Left unmanaged, every subsequent step in a session pays for everything that came before it, and the cost of a workflow grows quadratically in the number of steps while the quality of attention over that context degrades.

Compaction — summarising, pruning, and re-grounding as a session progresses — is therefore simultaneously a cost control and a quality control. Done well, it holds the working set small and relevant. Done badly, it discards the one detail the final answer depended on and produces a confident error that is very hard to trace, because the evidence was deleted before the mistake was made.

This is the entire subject of Chapter 36, and it is the reason the practice exam treats compaction and token budgeting as a first-class domain rather than an optimisation footnote. Teams that skip it discover that their cheapest model has become expensive purely through accumulated context.

You cannot route without evaluation. The decision to send a task class to a small model is only defensible if you can demonstrate that the small model is right on that class as often as the larger one, and the only way to demonstrate that is a graded set of representative tasks with known answers.

This is the least glamorous line item in an agentic budget and reliably the highest returning one. A harness of two hundred well-chosen cases lets you downgrade whole categories of work with evidence rather than hope, catch quality regressions when a provider silently updates a model, and prove to a sceptical finance function that a cost reduction did not purchase a quality reduction.

The argument for treating evaluation as core infrastructure rather than a testing chore is made in Chapter 17 and extended in the essay on why evals are the new dashboards. The short version is that an organisation without an evaluation harness cannot tell the difference between getting cheaper and getting worse.

Routing across model tiers is only possible if your context layer is not welded to one provider. If prompts, retrieval configuration, tool definitions, memory, and evaluation cases live inside a single vendor's proprietary surfaces, then every model decision becomes a migration and you will keep paying whatever the incumbent charges, because switching costs more than the difference.

The defensive position is straightforward and needs to be established before it is needed. Keep the definitions, retrieval logic, tool contracts, memory, and eval suite in systems you control, and treat the model as a replaceable component behind a boundary you own. This is the substance of Chapter 29 and Chapter 34, and it is the practical test set out in the essay on the exit test.

Cost discipline and portability turn out to be the same programme viewed from two angles. A team that can move a workload between models in an afternoon has permanent negotiating leverage. A team that cannot has a subscription with a rising price.

Start by picking the three agent tasks that carry the most volume and computing cost per correct answer for each one, including retries, escalations, and review minutes. The number will be uncomfortable and it will also be the first honest baseline your programme has had.

Then take the narrowest of those tasks and try it on the smallest model you have access to, with the context tightened rather than loosened: a canonical definition for every term in the prompt, a retrieval scope restricted to authoritative sources, a tool description that states exactly what the tool does and nothing more. Measure again. In most cases the small model with strong context beats the large model with loose context on the metric that matters, and it is faster.

Finally, write down the escalation rule and put a named human against each context asset the workflow depends on, in the sense described in the agent org chart. A routing architecture without accountable context is a cheaper way to be wrong at scale, and that is the one outcome worth more than the saving.