Cheaper Tokens Will Not Save Your AI Budget

Every few weeks another press release lands. A frontier model is cut in half. A mid-tier model is cut again. A small open-weights model arrives priced so low that finance asks, for the first time in any earnest way, whether the AI line item can finally stop growing. There is a real signal under the noise: raw inference is commoditising. Generations of hardware, competing APIs, and a maturing open-weights market are doing to the token what fibre did to the megabit. The decline is genuine, and over a long enough horizon it changes what is possible.

The problem is what the price war makes leaders assume. When the headline number on a unit falls, the instinct is to treat the whole category as cheaper. Procurement marks the line down. Teams that were told to be frugal with model calls start calling them freely. The mental model is the old one from cloud: the unit got cheaper, so the bill gets cheaper, and the work to do is simply to ride the curve down.

It does not work that way in agentic systems, and the gap between the assumption and the reality is where most of the surprise spend of 2026 has come from. A token is not a megabit. A megabit is a dumb pipe: send it, it arrives, the cost is the sending. A token is one step in a computation that may call itself, retry itself, retrieve context, branch, loop, and escalate. The price of one token tells you almost nothing about the price of one answer, and it tells you less than nothing about the price of one reliable answer at enterprise grade. Chapter 18Chapter 18 · 6 min LockedThe Hidden Cost of Agentic AIWhere the dollars actually go. opens the book's cost section by mapping exactly where the dollars go in an agentic call, and the first lesson is that the model fee is rarely the largest line.

This essay is a procurement and architecture argument, not a pricing forecast. The goal is to give a data or engineering leader the language to look at a falling price curve and ask the question that actually matters: does my total cost fall with it, or does it rise?

Denominators are quiet tyrants. They decide what a number means. Price per million tokens is a clean, comparable, spreadsheet-friendly denominator, which is why every vendor prints it and every dashboard tracks it. It is also the denominator of an era that is ending — the era of the single call.

Consider what a megabit-denominated bill rewards: moving fewer bytes over a cheaper pipe. Now consider what a token-denominated bill rewards: calling a cheaper model, fewer times. The first is a complete picture of the cost of a transfer. The second is a complete picture of nothing. It omits how many calls a single user request becomes, how much context each call carries, how many calls are retries, how many are escalations to a more expensive model after a cheap one failed, and how many are the orchestration layer reasoning about what to do next rather than doing it.

A team that optimises the denominator it can see — pick the cheapest model, watch the per-token rate trend down — will look virtuous on the dashboard and be quietly insolvent on the invoice. The honest denominator for an agentic system is cost per successful outcome: the dollars spent to produce one answer a user can act on without a human fixing it. That number is the one finance should be watching, and it is the one almost no dashboard surfaces, because it requires knowing what success means — which is the context problem the rest of the book is about. Chapter 21Chapter 21 · 6 min LockedQuality, Speed, and Cost TradeoffsHow to balance accuracy, latency, and spend. makes the quality-speed-cost tradeoff explicit, and the uncomfortable truth is that the cheapest path on the token axis is rarely the cheapest path on the outcome axis.

There is a second, subtler failure in the per-token frame. It treats the model as the product and everything else as overhead. In an agentic system the model is one component of a pipeline, and the pipeline is the product. Pricing the component and ignoring the pipeline is like pricing a factory by the cost of steel and being surprised the car is expensive.

Pull apart a real agentic bill and the model fee is usually the third or fourth largest line. The lines above it are the ones the price war does not touch, and in some cases makes worse.

The first is orchestration. Every branch, every tool selection, every plan-revise-check loop is a call. A user request that a human would answer in one turn can become eight, twelve, twenty calls in an agent that reasons out loud. Each of those calls is billable, and each carries context, so the later calls are more expensive than the earlier ones even at the same per-token rate. The price war drops the rate; the loop multiplies the calls. Multiply beats divide.

The second is retries. An agent that is confident but wrong does not fail cheaply. It produces an answer, the answer is checked — by a validator, by a human, by a downstream system — it is rejected, and the request runs again, often with more context appended about what went wrong. The retry is a second full journey through the loop, and it is the failure mode that turns a cheap model into an expensive one. A model at a quarter of the price that needs three attempts to be right is not a quarter of the cost. It is seventy-five percent more, plus the latency tax of doing it twice. Chapter 15Chapter 15 · 6 min LockedGuardrails, Approvals, and Audit TrailsDesigning safe agent behavior in practice. covers the audit and approval surface, and the practical point is that every guardrail you add to catch a wrong answer is also a potential retry you have to budget for.

The third is context. This is the line that grows the fastest and that the price war actively inflates. As models get cheaper per token, the rational move is to feed them more context — longer histories, larger retrieved documents, richer instructions — because the marginal token is cheap. But cheap tokens are still tokens, the working set grows, and a working set that doubles doubles the input cost of every subsequent call in the loop. Context is the one cost that scales with usage, with history, and with the decision to use a cheaper model, all at once. Chapter 36Chapter 36 · 10 min LockedCompaction and the Token BudgetBudget tokens like money, compact history without dropping commitments, and detect degradation before a user does. treats the token budget as a first-class engineering problem precisely because it is the line that escapes the spreadsheet first.

The fourth is the control layer. Approvals, logging, evaluation, and human review are not free. They are people time, which is the most expensive unit in the system, and they are compute spent on checking rather than answering. A cheaper model that fails more often pushes more work into the human review queue, and a human reviewer is priced in dollars per hour, not dollars per million tokens. The price war has no lever on the most expensive component, and a poorly routed agent can shift cost out of the cheap column and into the unbounded one. Chapter 17Chapter 17 · 6 min LockedHuman in the Loop Still MattersWhen humans should approve, review, or fully own the decision. sets out when a human must be in the loop, and the cost implication is that every human-in-the-loop checkpoint is a place where the token-price logic breaks down entirely.

This is the part that strains credulity, so it is worth stating precisely. A cheaper model does not always raise total cost. A cheaper model used naively — routed to tasks it cannot reliably do, with no escalation, no budget guard, and no measurement of retries — raises total cost. The mechanism is simple and brutal.

The cheap model is assigned broadly because it is cheap. It fails on a fraction of the requests it should never have seen. Each failure is detected late, after context has been assembled and a full reasoning pass has run. The failure routes to a retry, often on the same model with a longer prompt, which fails again on the harder cases. Eventually it escalates to the frontier model, which now has to do the work plus undo the confusion of two failed attempts. The frontier call is the most expensive call in the stack, and the cheap model has just guaranteed that the expensive one runs on the worst possible input, with the largest possible context.

The result is a system that uses the frontier model more, not less, than a system that routed to it directly from the start. The cheap model is not saving money. It is a tax on the expensive one, paid in latency and in the worst-case inputs. This is the core of Chapter 19Chapter 19 · 6 min LockedNot Every Task Needs the Best ModelRouting, cascades, and right-sized intelligence.: not every task needs the best model, and the art is matching the model to the task such that the cheap model handles the cheap work and the expensive model never sees a request it did not need to see. Done well, routing lowers total cost. Done badly — routing by price alone — it raises it.

There is a budget version of the same trap. A team sets a generous monthly token cap, watches the cheap-model spend stay well under it, and declares victory. The cap is on the wrong number. The cost that is growing is the human review backlog, the retry volume, and the escalation rate, none of which are on the token cap. Chapter 20Chapter 20 · 6 min LockedBudget-Aware AI DesignDesigning for budgets from day one. is about designing for the budget from day one, and the first design decision is which budget — token, call, outcome, or human-hour — is the one that will actually bind.

The way out is not to ignore the price war; it is to instrument the system so the price war helps instead of hurts. That requires a small, repeatable contract between the team that builds agents and the team that pays for them. It has four clauses, and any team can write the first version this quarter.

Clause one is the outcome denominator. Finance tracks cost per successful outcome, not cost per token. Define success for each agent — a ticket resolved without reopen, a report generated without correction, an action taken without rollback — and divide total spend by successful outcomes. The per-token number stays on the dashboard as a diagnostic, not as the headline. This single change reframes every conversation about model selection, because a model that is cheap per token but expensive per outcome stops looking free.

Clause two is the routing policy. Every request enters a router that can send it to a small, mid, or frontier model based on signals the team has validated — query complexity, required citation, permitted action scope. The router is not a guess; it is an evaluated artefact with a measured escalation rate. The policy's job is to ensure the frontier model sees only the requests that justify its cost, and the small model sees only the requests it can resolve. Chapter 19Chapter 19 · 6 min LockedNot Every Task Needs the Best ModelRouting, cascades, and right-sized intelligence. and Chapter 34Chapter 34 · 7 min LockedDesigning Multi-Agent Systems That Actually WorkWhen one agent is not enough — and how to make many cooperate without chaos. set out routing and multi-agent design, and the cost payoff is that the frontier line in the bill tracks the genuinely hard work, not the noise around it.

Clause three is the context budget. Every skill, every retrieved document, every conversation history has a token allowance, and the allowance is enforced, not advisory. When the working set approaches the budget, the system compacts — dropping low-value history, summarising, or escalating to a human — rather than quietly inflating every downstream call. This is Chapter 36Chapter 36 · 10 min LockedCompaction and the Token BudgetBudget tokens like money, compact history without dropping commitments, and detect degradation before a user does. made operational: budget tokens like money, and detect degradation before a user does. A context budget is what stops the cheapest-token strategy from becoming the most expensive one.

Clause four is the retry and escalation ledger. Retries and escalations are tracked as first-class cost events, with a reason code, so the team can see which cheap-model failures are paying for themselves in avoided frontier calls and which are paying for the frontier model's worst inputs. When the retry rate on a task climbs past its threshold, the routing policy moves that task up a tier. The price war becomes useful at this point: it gives the team room to move tasks down a tier when a cheaper model proves reliable, and the ledger is the evidence that proves it.

The contract is not a year-long programme. The first useful version is a week of work for a small team, and it changes the conversation with finance in the next review.

Start by picking one agent — the one with the most volume, not the one with the most glamour — and instrumenting its true cost per successful outcome for two weeks. You will likely find that the model fee is a minority of the spend and that the retry and escalation lines are larger than anyone assumed. That finding is the whole argument, in numbers your CFO already speaks.

Then stand up the router for that one agent. Two tiers is enough to begin: a small model for the easy classification and a frontier model for anything with action scope or required citation. Measure the escalation rate. If it is low, you have just demonstrated that the price war can lower your bill, on the one denominator that matters. If it is high, you have found the work the cheap model cannot do, which is more valuable still, because now you know where the frontier spend is actually earned. Chapter 19Chapter 19 · 6 min LockedNot Every Task Needs the Best ModelRouting, cascades, and right-sized intelligence. and Chapter 25Chapter 25 · 6 min LockedPlatform Independent, Platform AwareLearn the concepts. Map them to Databricks, Snowflake, AWS, Azure, GCP, and open source. cover the concepts and the platform mapping, so the same pattern moves cleanly across Databricks, Snowflake, and the hyperscalers.

Finally, put a context budget on that agent's working set and a retry ledger on its failures. These two are what turn the price war from a press release into a budget line that goes down. The token price will keep falling. Whether your total cost falls with it is a decision your team makes, not a curve your vendor draws.