The Blast Radius

Sit in enough agent design reviews and you notice that the same question opens almost all of them. How accurate is it? Someone produces a number. Ninety-one percent on the internal test set. Ninety-four after the last retrieval change. The room relaxes slightly, because a number has been produced, and a number feels like control.

It is the wrong question to open with. Not a useless question, but the wrong one to lead with, because it quietly assumes that the remaining nine percent is a rounding error rather than a set of real events that will happen to real records on a real Tuesday. Accuracy tells you how often the system is right. It tells you nothing at all about what happens the times it is not.

The better opening question is smaller and much harder to answer with a slide: when this agent is confidently wrong, what exactly can it touch, how far does the damage travel, how quickly do we notice, and how do we put it back? That is the blast radius. Every mature engineering discipline sizes it before shipping. Aviation sizes it. Finance sizes it. Database migrations size it. Agent programs, for the most part, still do not, which is why the industry keeps producing pilots that pass review and incidents that surprise everyone.

This is the Control pillar of the 4 C's, and it is the one most often reduced to a compliance checkbox. Chapter 13Chapter 13 · 6 min LockedGovernance Was Built for Humans. Agents Need More.Access control is not action control. makes the underlying argument: governance was built for humans reading things, and agents do things. The distance between those two verbs is the entire subject of this essay.

Here is why the distinction matters so much in practice. Accuracy is largely inherited. It comes from the model, the retrieval quality, the definitions, the prompt, and the difficulty of the task. You influence it, you improve it slowly, and you never fully own it. When the provider ships a new version, some of your accuracy changes without you touching a line of code.

Blast radius is entirely yours. Nobody else decides which tools the agent can call, which of those tools write rather than read, how many records a single run may touch, whether a run can be stopped mid-flight, whether the change is reversible, and who is paged when the counters move. Those are architecture decisions made by your team, in your codebase, on a normal working day.

That asymmetry should reorganize where the effort goes. Teams pour months into pushing accuracy from ninety-one to ninety-four, which is worthwhile, and spend an afternoon on containment, which is not. The uncomfortable arithmetic is that a three-point accuracy gain reduces the number of bad events by roughly a third, while good containment reduces the cost of every bad event, including the ones you have not imagined yet. One improves the odds. The other improves the outcomes across the whole distribution, and it keeps working when a model update quietly shifts behaviour underneath you.

This is also the honest answer to the executive who asks whether the agent can be trusted. Trust is not a property you assert about a model. It is a property you design into a system, which is the case Chapter 16Chapter 16 · 5 min LockedTrust Is Designed, Not AssumedHow leaders earn confidence in AI systems. makes at length.

Sizing blast radius does not require a risk taxonomy with forty categories. Two axes carry most of the weight. The first is reach: how many things does one action touch? A single draft reply reaches one customer. A bulk field update reaches every record that matches a filter, and filters are exactly the kind of thing an agent gets subtly wrong. Reach is the multiplier on any mistake.

The second axis is reversibility: how hard is it to undo? Reversibility is not binary, it is a cost with a clock attached. A database write with a version history is cheaply reversible for as long as the history is retained. A refund is reversible on paper and expensive in practice. An email to twelve thousand customers, a message posted in a public channel, a payment sent, a ticket closed with a customer notification, a record deleted from a system without soft deletes — those are not reversible at all. The action left your boundary and became someone else's memory.

Plot any agent capability on those two axes and the appropriate control falls out almost mechanically. Narrow reach and cheap reversal means run it autonomously and log it. Wide reach and cheap reversal means run it autonomously but sample the output, alert on volume anomalies, and keep the restore path warm. Narrow reach and expensive reversal means require approval on the action itself. Wide reach and expensive reversal means a human owns the decision and the agent prepares the work, which is the pattern Chapter 17Chapter 17 · 6 min LockedHuman in the Loop Still MattersWhen humans should approve, review, or fully own the decision. develops in detail.

The value of the exercise is not the grid. It is that the grid forces someone to say out loud which quadrant a capability sits in before it ships, in a room where a person with budget authority hears the answer.

Almost every team underestimates reach on the first pass, and the reason is consistent. The architecture diagram shows the tools the agent calls. It does not show what those tools call next. An agent that can update one field in the customer record has, in effect, whatever reach that field has downstream. If a status change triggers a lifecycle email, a billing recalculation, and a partner webhook, then the agent's true reach is three systems and one external company, not one field.

Second-order reach is where the genuinely embarrassing incidents live. The agent did exactly what it was permitted to do. The permission was scoped to a system that was itself unscoped. Nobody was wrong, and the customer still received four apology emails at two in the morning.

So map reach transitively, at least one hop past the boundary you think you own. For each write tool the agent holds, ask what fires when this changes, who receives a notification, what recalculates, what syncs outward, and what a downstream consumer will treat as a human decision even though a machine made it. The last one matters more than it sounds: once an agent's output enters another system, that system usually cannot tell the difference between a considered human judgement and a plausible generation.

There is a context dimension here too, and it is easy to miss. Reach is partly determined by what the agent can see. An agent with a retrieval scope covering every customer can, given the wrong instruction, act on every customer. Narrowing what enters the working set is a containment control as much as a quality control, which is one reason Chapter 35Chapter 35 · 11 min LockedRetrieval Mechanics: Chunking, Hybrid Search, and RerankingThe engineering layer under every context strategy — chunking, hybrid search, reranking, and how to prove it works. treats retrieval scope as an engineering decision rather than a search-quality preference.

Once reach and reversibility are honest, containment becomes concrete. Every consequential action should pass through five gates, and each gate should be a real piece of code rather than a paragraph in a policy document.

Scope comes first. The action carries an explicit boundary before it runs: which records, how many, within what time range, under whose authority. A tool that accepts an open-ended filter has no scope gate, and a limit of ten thousand is not a scope gate either — it is a comfort blanket with a large number written on it. Scope should be as tight as the task genuinely requires and should fail closed when the count exceeds expectation.

Simulate comes second, and it is the gate teams skip most often. Before writing, produce the diff. These forty-two records will change from this to that. This message will be sent to these recipients. This is the amount that will move. A simulation step is cheap to build, it makes approval meaningful instead of ceremonial, and it turns a class of catastrophic errors into an obviously wrong preview that somebody cancels in three seconds.

Approve comes third, and only where the quadrant demands it. Approval fatigue is a real failure mode: an approval queue that fires on everything trains humans to click through everything, which is worse than no approval at all because it manufactures the paperwork of oversight without the substance. Approve the expensive-to-reverse and the wide-reaching. Log the rest.

Execute comes fourth, and it should be idempotent, resumable, and interruptible. An agent action that cannot be safely retried will be retried anyway, by a queue, at the worst possible moment. Undo comes fifth, and it is the gate that separates a serious program from a demo. Before the action runs, the system should already know how it would be reversed and should have written down what it would take. Chapter 15Chapter 15 · 6 min LockedGuardrails, Approvals, and Audit TrailsDesigning safe agent behavior in practice. walks through guardrails, approvals, and audit trails as an ordered stack; the five gates are that stack made operational for a single call.

Two numbers describe your containment better than any policy: how long it takes to notice that an agent is misbehaving, and how long it takes to stop it once you have noticed. Multiply them by the action rate and you have the realistic worst case for a single incident. Most teams have never calculated this, and the calculation is usually sobering. An agent processing four hundred actions an hour, with a detection lag of six hours because the signal is a customer complaint, and a stop procedure that requires finding the right engineer, has a worst case in the thousands of actions.

Detection lag collapses when you monitor the right thing. Not model quality — volume and shape. Actions per hour against the normal band. Rate of a specific action type. Distribution of affected records. Repeat actions on the same entity. Rejections at the scope gate. Sudden silence, which usually means something upstream broke and often precedes a flood when it recovers. These are the same instincts a good operations team already applies to a batch job, and they transfer directly.

Time to stop collapses when stopping is a designed feature rather than an improvisation. A kill switch should be a flag that any on-call engineer can flip in seconds without a deploy, it should halt in-flight work at the next safe boundary rather than mid-transaction, and it should be scoped so you can disable one tool or one agent instead of taking down the whole assistant. If your only stop is a deploy, your stop time is your deploy time, and you should write that number on the risk register honestly.

The failure mode nobody plans for is the recovery flood. You stop the agent, fix the cause, and turn it back on to find a large queue of pending work that immediately executes at once. Resume should be gradual by default, and the resume path deserves as much design attention as the stop path.

A containment design that has never been exercised is a hypothesis. The only way to know whether the kill switch works is to pull it, on purpose, in production, on a scheduled afternoon when everyone is awake and nobody is surprised.

The drill is straightforward. Pick a real agent. Announce the window. Have someone flip the switch. Measure how long it actually took from the decision to the last action landing. Then reverse a batch of the agent's recent work using the documented undo path, not using a database administrator with a query editor and good intentions. Time that too. Write down what broke, what was missing, who could not be found, which runbook link was stale, and which restore step existed only in someone's head.

Teams that do this the first time are almost always humbled by it, and the specific failures are wonderfully consistent. The switch existed but required a deploy. The switch worked but left in-flight jobs running. The undo path covered the primary system and forgot the downstream sync. The audit log recorded that an action happened but not enough to reconstruct the prior state, which means it was a receipt rather than a restore point. Every one of those is cheap to fix once discovered and expensive to discover during a real incident.

Make the drill quarterly, put the two numbers in the same review that discusses accuracy, and treat a regression in them exactly as you would treat a regression in quality. That is also the honest version of the readiness conversation in Chapter 32Chapter 32 · 6 min LockedThe 4 C's Readiness AssessmentA full scoring rubric you can run on any AI project or program. — not a self-assessment, an exercise with a stopwatch.

The objection to all of this is predictable and worth taking seriously: does containment not slow everything down? If every action needs scoping, simulation, approval, and an undo path, has the agent not been reduced to an expensive form of automation with extra ceremony?

The opposite is true, and this is the part leaders tend to discover late. Containment is what makes autonomy politically possible. The reason so many agent programs stall at pilot is not that the model underperformed. It is that nobody could sign the risk. A capable agent with no containment story asks an executive to bet their credibility on a probability estimate, and sensible executives decline that bet. A slightly less capable agent with a tight scope, a live kill switch, a rehearsed undo, and a measured stop time is easy to approve, because the downside has been bounded and the bound has been tested.

That is why containment work usually expands autonomy rather than restricting it. Once the boundary is real, the sensible next question stops being whether to let the agent act and becomes how far to widen the scope this quarter. Teams with good containment ship more agent surface area, not less, and they ship it faster, because each expansion is a small adjustment to a known boundary rather than a fresh argument about trust.

There is a cost dimension too. Contained systems fail cheaply, and the cost of being wrong is a real line item that rarely appears in the model comparison spreadsheet — the argument developed in the cost of being wrong and in Chapter 18Chapter 18 · 6 min LockedThe Hidden Cost of Agentic AIWhere the dollars actually go. on where the money actually goes. A single uncontained incident routinely costs more than a year of the routing optimisations everyone spends their time on.

Start with an inventory that fits on one page. List every tool your agents can call. Mark each one read or write. For the writes, note the maximum number of records a single call can touch, whether the effect leaves your boundary, and how the change would be reversed. Most teams find at least one tool that nobody realised was a write, and at least one whose reach extends further than the person who added it believed.

Then place each write on the two axes and assign the control the quadrant implies. Be strict about the expensive-to-reverse cases. Anything that sends a message to a human outside the company, moves money, deletes without a recovery window, or changes a legal or contractual state belongs behind a simulate-and-approve gate until you have evidence to relax it.

Then measure the two numbers. Time to notice, time to stop. If you cannot measure them, that is your finding and it is more valuable than another point of accuracy. Finally, schedule the drill and invite the people who will be on call, because the point is to build the muscle in the group that will need it, not to produce a document.

None of this requires a platform purchase or a new framework. It requires treating agent actions the way every other consequential system in the company is already treated: bounded, observed, reversible, and rehearsed. The teams that reach real autonomy in the next two years will not be the ones with the best model. They will be the ones who made being wrong cheap enough to survive, often enough to learn from. Chapter 14Chapter 14 · 7 min LockedFrom Access Control to Action ControlMoving from who can see to what can act. on moving from access control to action control and Chapter 30Chapter 30 · 7 min LockedThe Trusted Agent ArchitectureA nine-step end-to-end reference architecture for production agents. on the trusted agent architecture are where that work becomes a reference design, and Chapter 34Chapter 34 · 7 min LockedDesigning Multi-Agent Systems That Actually WorkWhen one agent is not enough — and how to make many cooperate without chaos. extends it to the multi-agent case, where blast radius compounds across handoffs.