Semantic Drift: Why Green Pipelines Produce Broken Agents

Consider an illustrative Monday. A renewal agent recommends discounts for forty accounts it classifies as at risk. The pipeline that feeds it ran on time. Every dbt test passed. Freshness is under an hour. The observability dashboard is a wall of green. By Wednesday, the account team discovers that half of those customers were never at risk at all. Nothing broke. Something moved.

What moved was a definition. Two weeks earlier, the finance team changed what counts as an active account: trials no longer qualify, and accounts in a paused billing state now do. The column was still called is_active. It was still a boolean. Its null rate did not change. Its row count shifted by less than two percent, well inside every alert threshold. Every structural test was right to pass. The meaning was wrong, and the agent had no way to know.

This is semantic drift: the movement of business meaning underneath stable structure. It is not new. Analysts have always dealt with it, usually by noticing that a number looked strange and asking someone. The difference now is that the consumer is an agent, it does not find numbers strange, and it acts at a volume no reviewer can inspect. Chapter 6 argues that business meaning beats raw retrieval. Semantic drift is what happens when that meaning is allowed to change without anybody telling the system that reads it.

The modern data stack is very good at one kind of correctness. Data contracts verify that a column is a timestamp, that an identifier is unique, that a foreign key resolves. Freshness monitors verify that the table updated. Volume monitors verify that roughly the expected number of rows arrived. These are syntactic checks. They answer the question: does the data have the right shape?

Agents need a second kind of correctness. They need to know whether a field still means what the instructions, the examples, and the retrieved documentation say it means. Is revenue still recognised on the same basis? Does region still map to the same sales territories after the reorganisation? Does priority one still mean a customer-facing outage, or did support quietly start using it for VIP tickets? These are semantic questions, and almost no pipeline asks them.

The gap matters more for agents than it ever did for dashboards. A dashboard is read by a person who carries context: they know about the reorganisation, they heard about the pricing change, they remember that the March numbers are weird. Chapter 3 makes the point that smart models still need smart systems. An agent reads only what it is given. If the meaning changed and the context did not, the agent is reasoning correctly from a false premise, which is the most dangerous kind of error because it looks exactly like a correct one.

Semantic drift does not arrive one way. In most organisations it comes through four recognisable vectors, and each needs a different defence.

**Vector one: metric mutation.** A business definition changes and the field that carries it does not. Active customer, net revenue, qualified lead, churned account: every one of these has been redefined in most companies at least once, usually by a team with a good reason and no obligation to tell downstream consumers. The tell is a change in a definition document, a finance memo, or a semantic layer commit that has no matching change in the agent context. Chapter 7 explains semantic layers and metrics in plain terms; the short version is that a metric with no version is a metric that can drift silently.

**Vector two: the default-value trojan.** An upstream system adds a new category and routes it to an existing fallback. A new product line lands in other. A new refund reason is coded as general. A new contract type maps to standard. The schema is unchanged, the distribution shifts slightly, and an agent that was told other is rare and can be ignored starts ignoring a growing share of the business.

**Vector three: policy divergence.** The written policy says one thing; the practised policy says another. An executive approves an exception in a chat thread. Legal relaxes a rule for one region. Support adopts a workaround that becomes standard practice. None of it reaches the documents the agent retrieves. Chapter 11 calls context a living layer, not a document, precisely because the document is always behind the practice.

**Vector four: temporal anchor decay.** Context written in relative time stops being true without changing a word. Recent customers, the new pricing, last quarter's target, the current escalation path: each phrase was accurate on the day it was written and wrong some unknown number of days later. Embedded in a vector index, these phrases keep matching queries long after their anchor has passed.

It would be comforting if retrieval-augmented generation solved this. It does the opposite. Retrieval rewards similarity, and stale context is usually the most similar context available, because it was written in exactly the vocabulary people still use. A document describing the old definition of active account will match a question about active accounts beautifully. Its similarity score says nothing about whether it is still true.

Worse, retrieval adds confidence. A model asked a question with no supporting context will often hedge. The same model handed a well-written, highly relevant, slightly obsolete paragraph will answer crisply and cite it. The citation makes the answer look grounded. The grounding is in the past. Chapter 35 covers chunking, hybrid search, and reranking; none of those techniques can rank on truth if truth is not represented in the index as metadata the ranker can see.

Then the agent acts, and the action becomes new data. A discount granted on a drifted definition becomes a row in the deals table. A ticket routed on a drifted priority becomes training data for the next routing model. Drift that started as one stale paragraph turns into a feedback loop, each cycle reinforcing the last. This is why semantic drift in agentic systems is not a static error rate. It compounds.

The defence is to test meaning the same way we learned to test structure: explicitly, automatically, and before the consumer acts. Call these invariant assertions. An invariant is a statement about the business that must remain true for the agent's instructions to remain valid.

Good invariants are concrete and checkable. Active accounts exclude trials. Every order has exactly one recognised revenue date. Priority one tickets are customer-facing outages and represent less than five percent of volume. Discounts above twenty percent require an approval record. The region field maps to one of the seven current territories. Each is a sentence a business owner would agree with, and each can be expressed as a query.

The pattern has three parts. First, every invariant has a named owner, the person who would know if it stopped being true. Chapter 13 argues that governance built for humans is not enough for agents; ownership of meaning is where that gap is most visible. Second, invariants run on a schedule against production data, and also on demand when an upstream definition changes. Third, and most important, a failing invariant does not just raise an alert. It changes what the agent is allowed to do.

That last part is the difference between monitoring and control. Chapter 14 describes the move from access control to action control. An invariant gate is action control applied to meaning: if the assertion behind a workflow fails, the agent holds instead of acting, routes to a human, or falls back to a safer, narrower behaviour. The pipeline can stay green. The agent simply stops trusting it until a person confirms the meaning.

Invariants catch the drift you anticipated. You also need instruments for the drift you did not. Three practices cover most of the remaining surface.

**Shadow evaluation on canonical questions.** Keep a set of fifty or so golden questions whose correct answers a business owner has signed off, and rerun them daily against the live agent. When answers change and nobody changed the agent, something underneath moved. Chapter 29 treats evaluation as the backbone of trust; a golden set that only runs at release time misses exactly this class of failure.

**Decision divergence scores.** Compare what the agent recommends today with what it recommended for similar cases a month ago. A sudden shift in the share of accounts flagged at risk, tickets escalated, or invoices held is not proof of drift, but it is the cheapest possible signal that something deserves a human look.

**Reconciliation cadences with named owners.** Once a month, the person who owns each critical definition reviews the agent context for that definition against current practice. It takes an hour. It is the only control that catches policy divergence, because policy divergence is by definition the part that was never written down. Chapter 9 describes the context engineer; this review is one of the most valuable hours in that role's month.

Most drift is not caused by bad writing. It is caused by context that cannot say how old it is or who stands behind it. Fix that at the source and retrieval gets far safer.

Give every piece of agent context four fields: an owner, a valid-from date, a review-by date, and a definition version. Rank and filter on them. Context past its review date can still be retrieved, but it should be labelled as unverified in the prompt, and it should never be the sole basis for an irreversible action. Replace relative time with absolute time wherever possible: not recent customers, but customers who signed after the first of September.

This is the same idea Chapter 8 describes as the move from data catalogue to business memory. A catalogue lists what exists. A business memory knows what is currently true, who says so, and until when. Agents need the second.

The hardest part is not technical. Semantic drift lives between teams. Finance changes the definition; data engineering owns the pipeline; an AI platform team owns the agent; a business unit owns the outcome. Each can reasonably say the problem belongs to someone else.

The organisations that handle this well make one structural decision: a change to a business definition that agents depend on is treated like a change to an API that customers depend on. It needs notice, a version, and a migration window. Teams that change meaning publish the change. Teams that consume meaning subscribe to it. Chapter 15 covers guardrails, approvals, and audit trails; a definition change log is one of the simplest audit trails a company can keep, and one of the few that prevents incidents rather than explaining them afterwards.

If you are building this from scratch, start with the five definitions that drive the most agent actions. You do not need a company-wide semantic programme to stop the worst drift. You need five owners, five invariants, and one rule: no silent changes.

Use this as a Monday-morning diagnostic for any agent that acts on business data.

**1.** List the five business definitions your highest-volume agent depends on most.

**2.** Name one owner for each, someone who would know if it changed.

**3.** Write at least one invariant per definition as a plain sentence, then as a query.

**4.** Run those invariants daily, and wire failures to hold or narrow the agent's actions, not just to an alert.

**5.** Add owner, valid-from, review-by, and version fields to the context your agent retrieves.

**6.** Replace relative time phrases in agent context with absolute dates.

**7.** Keep a golden question set signed off by business owners and rerun it daily.

**8.** Track decision divergence on your agent's top three actions week over week.

**9.** Require that changes to agent-critical definitions are announced, versioned, and given a migration window.

Reader question: if finance changed the definition of an active customer this morning, how long would it take for your agents to find out, and who would tell them?