The Compute and Latency Budget — How Real Teams Cost Agentic Workflows

The first serious agent your team ships is not a product. It is an audit — of every unexamined assumption in your architecture, priced in dollars and milliseconds, and delivered thirty days later as a line item on a cloud invoice you cannot argue with.

The pattern is familiar. A prototype impresses a room. A pilot moves to production. Usage grows. Somewhere between week three and week eight, a finance leader forwards an invoice with a red circle around it, and the same engineering team that shipped the demo now has to explain, in front of people who have never seen a token, why an assistant that felt like magic costs more per query than a paid analyst.

The uncomfortable truth is that most agentic systems in production today were never budgeted. They were sized. Sizing is what you do when you know the load. Budgeting is what you do when you know the load, the alternatives, and the price you are willing to pay for each additional unit of quality. Very few teams have done the second one before shipping. This essay is a compressed version of the discipline that fixes that — the field version of Chapters 13 through 17 of The Context Advantage.

The single biggest analytical error in agent economics is to reason about cost per token. Token cost is the easy number, which is why everyone quotes it. It is also almost never the number that determines your bill.

The number that determines your bill is cost per successful task — the total spend across every model call, tool call, retry, reflection, and orchestration hop required to turn a user's request into an outcome they accept. That number is typically five to fifty times the naive one-shot token cost, and it is not linear in any parameter you can point at on a pricing page.

A trivial worked example. A support agent answers a question. The naive cost is one prompt, one completion — maybe 1,500 tokens in, 400 out, a fraction of a cent on a mid-tier model. The real cost path is a retrieval step (embed the query, vector search, rerank), a plan step (a reasoning model deciding what to do), two tool calls (one to a ticketing system, one to an order database), a synthesis step, a self-check step, and — on 8% of sessions — a human handoff that costs a full round of context re-injection. The real cost per resolved ticket is not 0.4 cents. It is 6 to 11 cents, depending on the day. The gap between those two numbers is the gap between a healthy AI product and a boardroom presentation about pausing the AI initiative.

The point of Chapter 13 is that this is not a measurement problem to solve after launch. It is a design constraint to hold from the first architecture diagram. Every arrow in that diagram is a cost. Every retry policy is a cost multiplier. Every 'just in case' context injection is a standing charge. If your architecture review does not draw the cost graph next to the data flow graph, you are shipping an invoice you have not read.

An agentic workflow has four budgets running in parallel, and if you do not set them explicitly, the runtime sets them for you in the worst possible way — with a stack trace, a timeout, or a surprise bill.

Budget one: token budget per task. The total input and output tokens you are willing to spend on a single user request across all hops. Not per call. Per task. The teams that get this right assign a token ceiling at intake and enforce it at the orchestrator, so an unusually chatty tool response cannot silently triple the spend.

Budget two: latency budget per task. The end-to-end wall-clock time from user submit to visible answer. This is the budget most engineering teams do think about, but they think about it as a performance concern, not a cost concern. It is both. Latency is the variable that decides whether you can parallelize tool calls, whether you can afford a reflection step, and whether a cheaper model with two retries beats a pricier one with none.

Budget three: tool-call budget. The maximum number of external tool invocations per task. Every tool call is a network round trip, a rate-limit consumer, and a potential failure mode. Agents without a tool-call ceiling love to enter three-tool loops that resolve nothing and bill everything.

Budget four: retry and reflection budget. Reflection is powerful and expensive. A single self-critique can add 40 percent to the cost of a task. Two can double it. If you do not decide, in advance, how many reflection rounds are allowed and under what conditions, you have not shipped an agent. You have shipped a research paper with a payment method attached.

Sizing starts with a target unit economic. Pick the metric that matches your product. Cost per resolved ticket. Cost per drafted contract. Cost per approved code review. Cost per completed onboarding step. Whatever it is, pick a number your business would be delighted to run at, a number it could tolerate, and a number that would kill the feature. Three lines. Publish them.

Then work backwards. If the delight number is 4 cents per resolved ticket and your architecture makes an average of three model calls and two tool calls per task, you have roughly 0.8 cents to spend per model call on average — which immediately tells you the largest model you can afford as your default and the smallest one you must route to for the easy majority. This is the routing conversation from Chapter 14, but with a concrete constraint that ends the argument.

Latency budgets are sized the same way, from the outside in. If a user will tolerate 4.5 seconds end to end and you have five hops in the critical path, you have under a second per hop on average — which tells you what can be sequential and what must be parallel, and it tells you exactly which model tiers are viable. A frontier model with a 2.8-second median first-token time is not eligible for a hop in a 4.5-second budget, regardless of how good its answers are, unless it is the only hop.

Tool-call and reflection ceilings are political as much as technical. Set them low enough that the majority of tasks finish inside them, and treat any task that hits the ceiling as an incident, not a feature. Ceiling breaches are the earliest and cheapest signal that a workflow is drifting toward the shape of a research demo instead of a product.

Once budgets are explicit, the choice between architectures stops being a religious argument and becomes an arithmetic one. There are, in practice, four common shapes, and each of them has a very different cost and latency profile.

One: a single-shot LLM call with grounded context. Cheapest, fastest, most predictable. Correct for a much larger share of enterprise use cases than the agentic hype cycle wants to admit. If a task can be solved by a well-retrieved prompt on a mid-tier model, the answer is almost always to solve it that way and move on. Chapter 6 on grounded context is the argument for why this is not a compromise; it is often the right architecture.

Two: a tool-augmented single-agent loop. A planner-executor over a small, well-typed tool set with strict ceilings on hops and reflections. Two to five times the cost of shape one, roughly two to four times the latency, but a much wider capability envelope. This is the sweet spot for most 2026 enterprise agents, and it is where cost discipline pays off fastest.

Three: a multi-agent orchestration with role specialization — a researcher, a critic, a synthesizer, a verifier. Powerful for open-ended tasks, catastrophic for cost if you do not cap coordination. A well-run multi-agent system runs at three to eight times the cost of shape two. A poorly-run one runs at fifteen to forty times, and the failure mode is agents politely paying each other to disagree in loops.

Four: an offline agent that runs asynchronously against a queue. Highest per-task budget, but no user-facing latency constraint, which unlocks larger models, longer reflection, and better verification. For the right workload — nightly research, weekly audits, batch code migration — this is the shape that quietly generates the most value per dollar in the entire stack, and it is the least talked-about at conferences.

The mistake is not choosing one shape. The mistake is choosing the same shape for every workflow. Mature platforms run all four in production and route each user request to the cheapest shape that can meet the quality and latency budget for that task. That routing layer — described in Chapter 14 — is the highest-leverage engineering artifact in a Cost-aware platform.

Once you can see the cost graph, the levers become obvious. In order of typical impact:

Lever one: model routing by task class. Send the easy 70 to 80 percent to a small fast model. Send the medium 15 to 25 percent to a mid-tier. Reserve the frontier for the last few percent that truly need it. Median observed savings when a team turns this on for the first time: 55 to 75 percent, at neutral quality. This is Chapter 14.

Lever two: aggressive prompt and context trimming. Most system prompts carry 40 to 60 percent dead weight — historical instructions, safety scaffolding copied from a demo, tool descriptions for tools this call will not use. Trim per call, not per assistant. Median savings: 15 to 30 percent, and often a latency win as a bonus.

Lever three: semantic and exact-match caching. A surprising share of production traffic is near-duplicate. An answer cache with a good invalidation policy commonly deflects 20 to 40 percent of calls on read-heavy workloads. Chapter 14 again, and it is the fastest ROI change in the book.

Lever four: batching and parallelization inside a task. Independent tool calls should never run sequentially. Independent sub-questions should be embedded and retrieved in a single batch. This is a latency win first and a cost win second, but the latency win often unlocks a smaller model choice, which is where the money is.

Lever five: reflection budgets with early exit. Reflect once when the confidence is low, skip reflection when the confidence is high, never reflect on cache hits. This one change routinely trims 20 percent off multi-agent bills.

Lever six: streaming and speculative decoding for user-facing hops. Streaming does not reduce cost, but it changes the perceived latency budget, which in turn lets you choose a slightly larger and better model without violating the user experience contract. That trade — a small cost increase for a large quality increase within a fixed perceived-latency budget — is almost always worth it.

Lever seven: async offload for anything that does not need to be synchronous. If a task can wait 30 seconds, it can run on a cheaper tier. If it can wait 30 minutes, it can run overnight on the cheapest tier available. The share of workflows that were made synchronous by accident, not by requirement, is very large.

None of the levers above matter without unit-cost telemetry. This is the Chapter 15 argument in one sentence: if cost per successful task is not on a dashboard next to latency and quality, cost is not a real constraint in your organization — it is a rumor.

The specific artifacts that separate the teams doing this well are unglamorous. A daily unit-cost report by feature. A budget field on every feature spec, with a named cost owner. An alert at 70 percent of monthly budget that pages the on-call, not a distribution list. A launch checklist that treats a cost regression the same way it treats a latency regression — as a launch blocker, not a footnote. These are Chapter 16.

The cultural piece — Chapter 17 — is the hardest and the most durable. FinOps for AI works when engineering, finance, and product share one metric and one dashboard, and when unit cost is discussed in the same review as reliability and quality. It fails, every time, when cost is a finance function reporting on decisions engineering already made.

If you take one thing from this essay into Monday, take this: pick your single highest-volume AI feature in production and compute its cost per successful task. Not per token. Per task. Publish the number. That single act starts more useful conversations than any architecture diagram in your backlog.

In week two, draw its cost graph. Every model call, every tool call, every retry, every reflection. Label each edge with its typical token count and typical latency. Nine times out of ten, one edge on that graph is doing 40 percent of the damage — an over-large default model, an unbounded reflection loop, an untrimmed system prompt, a synchronous tool that could have been async. The fix is usually a day of work and a multiple-month payback.

Then, and only then, decide whether the feature belongs in shape one, two, three, or four from the taxonomy above. Most features that were shipped as shape three quietly want to be shape two. Most features shipped as shape two quietly want to be shape one with better retrieval. This is not a downgrade. It is the definition of engineering maturity in the agentic era.

Where this lives in the book — direct links:

→ Chapter 13 — Cost as a Design Constraint: /context-advantage/book/chapter-13

→ Chapter 14 — Routing and Caching: /context-advantage/book/chapter-14

→ Chapter 15 — Unit Cost Telemetry: /context-advantage/book/chapter-15

→ Chapter 16 — Budgets and Alerts: /context-advantage/book/chapter-16

→ Chapter 17 — FinOps for AI: /context-advantage/book/chapter-17