The Model Router: Stop Hardcoding Intelligence

Search any agent codebase that has been in production for a year and you will find the same line repeated dozens of times: a model name, typed by hand, inside a function that also builds a prompt, calls a tool, and parses a reply. Nobody decided that this string should govern the company's AI costs, latency, and vendor exposure. It simply accumulated, one sprint at a time.

That string is now load-bearing. When the vendor raises prices, every service pays. When the vendor has a regional outage, every agent stops. When the vendor deprecates the version, a team spends a month finding every call site and re-testing every prompt that was tuned to its quirks. Chapter 22 calls this the danger of lock-in, and the uncomfortable truth is that most lock-in is not signed in a contract. It is typed into code.

The fix is not to pick a better vendor. It is to stop letting any single model name live inside business logic. That is the job of a model router: one thin layer that receives every request, decides which model should serve it, and hides the answer from the rest of the system.

Many teams believe they are already multi-model because they hold accounts with two or three providers. Ask a harder question: if your primary provider doubled its price tonight, how much of tomorrow's traffic could you move, and how long would it take? For most organisations the honest answer is very little, and weeks.

The reason is that the switching cost does not sit in the API call. It sits in everything around it: prompts tuned to one model's habits, tool schemas written in one vendor's dialect, output parsers that assume one formatting style, and evaluation results that were only ever run against one model. One Model Is a Single Point of Failure describes the outage risk. The router is the architecture that removes it.

The first design decision is to stop asking which model is best and start asking which model is enough. Chapter 19 makes the case plainly: not every task needs the best model. In practice, most agent workloads fall into three tiers.

The triage tier handles classification, intent detection, entity extraction, schema checks, and short rewrites. Small models do this well, quickly, and at a fraction of the price. The workhorse tier handles most agent turns: choosing tools, drafting SQL or API calls, summarising retrieved passages, and producing structured answers. Mid-sized models carry the bulk of production traffic here. The reasoning tier is reserved for the hard cases: multi-step plans with real consequences, conflicting evidence, verification of high-stakes outputs, and recovery when cheaper tiers fail.

A useful starting heuristic is that the reasoning tier should serve a small minority of calls, and every call that reaches it should be able to explain why. If the frontier model is answering password-reset questions, you are paying reasoning prices for triage work. Small Models, Big Context shows why a well-fed small model often beats a starved large one.

A router is only as good as the signals it reads. Four matter most, and each one maps to a pillar of the 4 C's.

Complexity is the Context signal. How many sources must be reconciled, how ambiguous is the question, how long is the assembled context? A cheap classifier, or simple rules on token count and intent, can estimate this before any expensive call is made. Pair it with the layout discipline in The KV Cache Discipline so stable context stays cached whichever tier serves it.

Budget is the Cost signal. Every request should carry a budget envelope: a ceiling on spend and a target on latency. Chapter 20 calls this budget-aware design, and Chapter 21 explains the quality, speed, and cost triangle the router is constantly negotiating.

Sensitivity is the Control signal. Requests that touch regulated or personal data may only go to models running inside your own boundary or to providers with the right agreements. This is a policy decision, not a performance one, and the router must enforce it before any other consideration. A cheaper route that leaks data is not cheaper.

Health is the Choice signal. If a provider's error rate or latency has crossed a threshold, the router should stop sending it new work. That is where the router meets the circuit breaker, covered below.

Routing solves which model answers. It does not solve the deeper lock-in, which lives in how you talk to the model. If your system prompts, tool definitions, and few-shot examples are written in one vendor's dialect, every route change becomes a rewrite.

The answer is a canonical context packet. Your application builds context in a neutral shape: the instruction, the business definitions, the retrieved evidence with its sources, the available tools described once, the output contract, and the permissions in force. An adapter then compiles that packet into each provider's format at the last moment, the way a compiler turns one source file into different machine code for different chips.

Chapter 12 describes this as portable context, an open contract for agents. Chapter 24 frames the same idea as building for change. Once context is neutral, adding a new model is an adapter task measured in days, not a migration measured in quarters.

Every router needs a plan for when its first choice fails. The simplest pattern is a cascade: try the preferred model, and on timeout, error, or a failed output check, retry once on a warm secondary. The important word is warm. A fallback that has never served real traffic will fail exactly when you need it, because its prompts were never evaluated and its quotas were never raised.

Wire the cascade to a circuit breaker, as in The Circuit Breaker Pattern for AI Agents. When a provider trips, the router sheds all new work to the secondary instead of letting each request discover the outage on its own. When the provider recovers, a small half-open trickle tests it before full traffic returns.

For latency-critical paths, some teams use hedged requests: if the primary has not answered within a set percentile, send the same request to the secondary and keep whichever answer arrives first. It costs extra tokens on a small share of calls and buys a much flatter tail. Use it where the latency matters more than the money, and nowhere else. Design for the Bad Day covers what the agent should do when every route is down.

A router makes switching easy, which makes switching dangerous. The guardrail is evaluation. Before any model receives production traffic for a task, it must pass the golden set for that task: the real questions, the expected answers, the acceptable sources, and the cases where it should refuse.

Move traffic in steps. Shadow first, where the candidate answers in parallel and nobody sees it. Then a small canary share. Then a larger share, with the evaluation scores, cost per resolved task, and complaint rate watched at each step. If any of them regress, the router rolls back by changing a weight, not by shipping code. The Evaluation Gap explains why demo scores are not enough to justify the move.

This is also how you win the procurement conversation. When a vendor offers a discount, you can answer with data within a day: here is how your model scored on our tasks, here is what it would cost at our volume, here is the share of traffic it has earned. That is choice made concrete.

The router quickly becomes one of the most important pieces of the stack, so treat it accordingly. Routing policy should live in versioned configuration with a named owner, reviewed like any other change. Every decision should be logged: which model was chosen, which signals drove the choice, what it cost, how long it took, and whether a fallback fired. Those logs are your audit trail, your cost report, and your incident timeline, all at once, as Chapter 15 recommends for any agent action.

Keep the router thin. It should route, adapt, fall back, and record. It should not hold business logic, retrieval code, or prompt craft. The moment it starts making product decisions, you have built a new monolith with a nicer name. Chapter 25 captures the balance: platform independent, platform aware. Use each provider's strengths through the adapter, and never let one of them become a dependency you cannot remove.

Finally, measure the right numbers on the router dashboard: cost per resolved task by tier, the share of traffic on each provider, fallback rate, tail latency, and evaluation score by route. If you can only show one chart to leadership, show how much traffic you could move within a week. That number is your real exposure. For a broader buyer's view, read The Exit Test and The AI Vendor Scorecard.