Why Context Compaction Beats Giant Token Windows in Production AI Agents

Consider an illustrative operations team running a reconciliation agent. On turn three it answered in two seconds and cost a fraction of a cent. By turn forty, after dozens of tool calls returning full query results, it took eighteen seconds per reply, each turn cost more than the whole first hour combined, and it had started ignoring the instruction never to post adjustments above a set limit without approval. Nothing was broken. The window was simply full of everything the agent had ever seen.

The team's first instinct was to buy a model with a bigger window. That made the problem cheaper to hit and more expensive to live with. The real fix was the opposite: decide what the agent actually needs to remember, keep that, and let the rest go. That discipline is called compaction, and it is the subject of Chapter 36.

Large windows fail in three ways at once. Latency grows because the model must process every token before it writes the first word of its answer. Cost grows faster than most teams expect, because every turn re-sends the entire history, so a fifty-turn task pays for the early turns fifty times. Chapter 17 treats this as a compute and latency budget, and an uncompacted agent blows through it quietly.

The third failure is the most dangerous: attention dilution. Models are measurably better at using information at the start and end of a long input than in the middle. A rule stated once on turn one and followed by forty tool dumps is technically present and practically invisible. The Context Degradation Curve described this decay; compaction is how you flatten the curve.

Naive fixes make it worse. Dropping the oldest turns first, the common default, throws away exactly what matters most: the original goal, the user's constraints, and the permissions the agent was given. The agent stays fast and becomes confidently off-task.

The compaction ratio compares the tokens an agent has consumed over a task to the tokens in its active working context. A healthy production agent might read two hundred thousand tokens of tool output over a long task while its working context stays between four and eight thousand. That is a ratio of roughly twenty-five to one or better.

The point is not the exact number. It is that working context should stay roughly flat as a task gets longer. If your context size grows in a straight line with turn count, you do not have a memory design. You have a transcript. Track working-context size per turn on the same dashboard as cost per task and answer quality, as Evals Are the Dashboards recommends.

Compaction becomes simple once context is split into tiers that are treated differently.

Anchors are pinned and never summarised: the goal, the hard rules, the user's identity and permissions, and the definition of done. They sit at the top of every request, which also keeps them cache-friendly, as The KV Cache Discipline explains. Only an authorised human or system can change them.

The ledger is structured state: confirmed facts as fields, not prose. Account identifiers, approved amounts, chosen options, decisions made and why. Each entry carries its source so it can be checked or retracted. This is the working memory Chapter 8 distinguishes from the long-term store.

Scratch is everything else: raw tool output, intermediate reasoning, retrieved passages for the current step. It is useful briefly and then it should leave the window, with a pointer to where the full record lives if anyone needs it later.

The tempting shortcut is to ask the model to summarise the conversation so far. Generic summaries are good at gist and bad at precision. They round numbers, merge similar identifiers, drop negative constraints such as do not contact this customer, and turn a tentative finding into a stated fact. Over several rounds of summarising summaries, small losses compound into the kind of error Context Poisoning describes.

Schema-guided extraction avoids this. Instead of asking for a summary, ask the model to fill a defined structure: which facts were confirmed, with what values, from which source, and what is still open. Validate the output like any other data, reject entries without a source, and keep identifiers exact. The ledger is smaller than a summary and far more trustworthy.

Run compaction as a step, not an afterthought, either every few turns or when working context crosses a threshold.

**Extract.** Pull confirmed facts and decisions from recent scratch into a structured form, with source pointers.

**Reconcile.** Merge them into the ledger. When a new fact contradicts an old one, replace it deliberately and record why, rather than keeping both and letting the model guess.

**Evict.** Remove raw tool output and stale retrieved passages whose useful facts are now in the ledger. Store the full record outside the window, so nothing is lost, only moved.

**Re-pin.** Restate the anchors at the top of the next request, unchanged. This is what stops long tasks from drifting off their original goal and rules.

Compaction decides what stays in the window; retrieval decides what comes back in. Evicted material should remain findable, so a well-designed agent can fetch the full record of an earlier step when a question needs it. Chapter 35 covers the retrieval mechanics that make this cheap and precise.

For multi-agent work, compaction is also the handoff format. A sub-agent should receive anchors plus the relevant slice of the ledger, not the parent's whole transcript. That fresh, small window is why delegation often beats one giant session, a pattern explored in The Handoff Problem.

Compaction is usually sold as an engineering optimisation. It is really a business control. Flat working context means predictable cost per task, which is what finance needs to approve scale, as Chapter 18 argues. Pinned anchors mean rules are still enforced on turn fifty, which is what risk teams need. And a sourced ledger means every decision can be traced, which is what auditors ask for.

It also protects choice. A compact, structured state is portable across models; a hundred-thousand-token transcript tuned to one vendor's window is not. Teams that compact well can route work to smaller, cheaper models, as Small Models, Big Context shows.