The KV Cache Discipline

A support agent looks inexpensive in a demo. One question, one answer, a few thousand tokens. Then it reaches production, where a real conversation runs twenty turns, the system prompt carries forty pages of policy, the tool catalogue describes thirty functions, and every turn re-sends all of it. The model does not remember the previous turn. It reads the whole transcript again, from the first token, every time.

Take an illustrative case. If the stable part of the prompt is sixty thousand tokens and the conversation adds one thousand per turn, turn twenty sends roughly eighty thousand input tokens. Summed across the session, the agent has paid for more than a million input tokens to hold one conversation. Multiply by a few thousand sessions a day and the line item stops being a rounding error. These numbers are a worked example, not a benchmark, but the shape is real: without caching, multi-turn cost grows faster than the conversation does.

Every major serving stack now offers a way out. Hosted providers discount cached input tokens heavily, and open serving engines such as vLLM and SGLang reuse computed attention state for shared prefixes. The discount is not automatic, though. It is earned by the shape of the prompt. That shape is a context engineering decision, which is why this belongs next to the arguments in Chapter 36 on compaction and the token budget.

When a transformer reads a prompt, each layer computes key and value vectors for every token. That intermediate state, the KV cache, is what lets the model produce the next token without re-reading everything. Prefix caching keeps that state after the request ends. When the next request begins with exactly the same tokens, the server skips the computation for that stretch and starts work where the shared prefix ends.

The word that matters is exactly. Caching matches from token zero forward, and the match stops at the first difference. Change one character on line three of the system prompt and everything after line three is recomputed at full price. A cache is not a semantic memory. It does not know that two prompts mean the same thing. It knows only whether they begin with the same bytes.

That single rule turns prompt layout into an economic decision. Content that rarely changes must come first. Content that changes every turn must come last. Anything in between should be ordered by how often it changes. Teams that internalise this rule pay for their stable context once per cache lifetime instead of once per turn.

The first habit is the timestamp at the top. Many system prompts open with the current date and time so the model knows what day it is. That one line, placed first, guarantees that no request ever shares a prefix with the one before it. The fix is trivial: move the time to the end, and round it to the day unless the task truly needs the minute.

The second habit is unstable tool definitions. Tool catalogues are often assembled from a dictionary or a database query whose order is not guaranteed, and serialised with keys in whatever order the library prefers. The model sees the same tools, but the bytes differ, so the cache misses. Sort tools by name, serialise with stable key order, and version the catalogue so it changes on purpose rather than by accident.

The third habit is splicing retrieval into the middle. Teams drop freshly retrieved passages between the system rules and the task instructions, because that is where they seem to belong logically. Every new retrieval then invalidates everything after it, including instructions that never changed. Retrieved evidence belongs near the tail, clearly labelled, as Chapter 35 recommends for reasons of grounding as well as cost.

The fourth habit is personal state before shared state. A user's name, preferences, or permissions placed ahead of enterprise-wide policy make every user's prefix unique, so no two users can share the expensive part. Shared material first, personal material after it, lets one cached policy block serve an entire organisation.

The discipline becomes practical when the prompt is assembled in tiers, each with a known rate of change. Tier one is the immutable prefix: identity, safety rules, output contracts, and the frozen tool catalogue. It should change only on a release, and it should be the longest stretch of identical bytes your system produces.

Tier two is slow-moving organisational context: the glossary, current policies, product catalogue, and the curated institutional memory described in Chapter 10. It changes daily or weekly on a schedule. Because it changes on a schedule, its cache cost is predictable, and it can be refreshed deliberately, the way Chapter 11 argues a living context layer should be.

Tier three is the session log. Each turn is appended, never rewritten. When the conversation grows too long, compaction replaces an old block with a summary at a fixed boundary rather than rewriting the whole history, so the prefix before that boundary stays intact. Tier four is the ephemeral tail: the current question, fresh retrieval results, the latest tool outputs, and the current time. It changes every turn, and because it is last, changing it costs nothing upstream.

This ordering also improves the quality of answers, not only the price. Stable instructions in a fixed position behave more consistently, and labelled evidence at the tail is easier for the model to cite. Cost discipline and context discipline turn out to be the same habit seen from two sides.

Cached input is typically billed at a small fraction of the normal input price; on several hosted platforms the discount is around ninety percent, though writes to the cache may cost a little more than ordinary input. Exact rates vary by provider and change often, so treat the following as arithmetic rather than a price sheet.

Suppose normal input costs three dollars per million tokens and cached input costs thirty cents. A session in which eighty-five percent of input tokens hit the cache pays an effective rate of roughly seventy cents per million, less than a quarter of the uncached price. The curve of cumulative cost across a long conversation stops climbing steeply and starts to flatten.

That is why the right metric is not price per token but cost per correct outcome, the measure Chapter 15 builds the economics of context around. A cache hit rate is a lever on the denominator you already care about. Report it next to accuracy and cost per resolved task, and the conversation with finance changes from 'tokens are expensive' to 'we are paying full price for text we already sent'.

Finance notices the invoice. Users notice the wait. Processing a long prompt before the first token appears is often the largest share of perceived latency, and a cached prefix removes most of it. On long prompts, time to first token commonly drops by half or more when the prefix is reused; the exact gain depends on the model, the hardware, and how long the prefix is.

For agents, this compounds. A planning agent that calls six tools in sequence makes seven model calls, each of which re-reads the growing context. Shave a second from each and the user waits seven seconds less for a single task. That is the difference Chapter 17 draws between a latency budget that is designed and one that is merely endured.

There is a resilience benefit as well. Caches expire after minutes of inactivity on most hosted platforms. Teams that understand this schedule warm critical prefixes before peak hours and route related requests to the same serving replica, rather than discovering at nine in the morning that every first request of the day is slow.

Caching is not free of risk. The most common failure is freezing something that should change: a policy is updated, but an engineer pins the old text into tier one to protect the hit rate, and the agent keeps citing the retired rule. The cost of being wrong, as Chapter 16 puts it, is always larger than the cost of a token. Correctness wins every tie.

The second failure is tenant leakage by design. A shared prefix should contain only what every caller is allowed to see. Placing one customer's documents in a prefix that another customer's requests reuse is not a caching problem but a permission problem, and it needs the same gates described in Chapter 9. Providers isolate caches by account, but your own application can still mix tenants if it builds the prefix carelessly.

The third failure is optimising the metric instead of the outcome. A team can push its hit rate to ninety-nine percent by refusing to refresh context at all. The goal is not the highest hit rate. It is the cheapest correct answer, delivered fast, from context that is still true.

Begin by measuring. Most providers report cached and uncached input tokens per request; log both, and compute the hit rate per agent and per route. Many teams find their first number is close to zero, which is good news, because the fix is usually a reordering rather than a rewrite.

Next, diff two consecutive requests from the same session byte by byte and find the first difference. That position is your effective cache boundary. Move whatever sits there, often a timestamp, a request identifier, or a shuffled tool list, to the tail. Repeat until the first difference sits where the session log begins.

Finally, write the tier order into your prompt-assembly code as a contract, with a test that fails if a volatile field appears in tiers one or two. Cache discipline that lives only in one engineer's head disappears at the next refactor. Cache discipline enforced by a test survives.