Do Not Trade Open Data for Closed AI

For most of the last decade, the central architectural fight in enterprise data was about formats. It was a fight about whether your tables belonged to you or to the engine that read them. It was won, more or less, and it was won in public: open table formats, open file formats, open catalogs, engines that compete for the right to read data they do not own. The practical result is that a large organisation today can change its query engine without changing its data, and that is an enormous amount of freedom compared with 2015.

Then agentic AI arrived, and something quietly happened that almost nobody put on an architecture diagram. The lock-in did not go away. It moved up one floor. It left the storage tier, where everyone was watching, and settled into the intelligence tier, where almost nobody is.

The storage tier is where your rows live. The intelligence tier is everything that makes those rows usable by a machine that reasons: the definitions, the semantic mappings, the retrieval indices, the tool descriptions, the evaluation suites, the memory of what worked. That layer is where the last five years of your organisation's hard-won meaning now accumulates. And in most enterprises it is accumulating inside a vendor's product, in a shape nobody outside that product can read.

This is the Choice pillar, applied to the newest and least visible part of the stack. Chapter 22Chapter 22 · 5 min LockedThe Danger of AI Lock-InHow yesterday's choices limit tomorrow's options. describes the mechanics of AI lock-in; Chapter 23Chapter 23 · 5 min LockedOpen Formats, Open Interfaces, Open ThinkingWhy openness is a strategic moat. makes the case for open formats, open interfaces, and open thinking. The argument of this essay is narrower and more urgent: you can have completely open data and still be completely captured, and most organisations are currently on exactly that path.

The migration was invisible for three reasons, and all three are reasonable.

First, it did not feel like a platform decision. It felt like enabling a feature. Nobody convened an architecture review to approve a semantic model, a set of tool definitions, or a retrieval index. Those things arrived as configuration inside a product the organisation had already bought. A checkbox is not a contract negotiation, so no negotiation happened.

Second, it was genuinely good. Vendors who own both the storage and the intelligence tier can do things an assembled stack cannot: faster retrieval, tighter permissions, better defaults. The integration is real value, not a trick. That is precisely what makes it dangerous — the lock-in arrives attached to something that works.

Third, the cost of leaving is invisible until you try. Data migration has a known shape: you can estimate the tables, the volume, the cutover window. The cost of moving an intelligence tier is nowhere on a spreadsheet, because nothing in it looks like an asset. It looks like settings. Until the day you try to run the same agent somewhere else and discover that eighteen months of accumulated organisational meaning does not come with you.

Lock-in at the intelligence tier is not one thing. It is four, and they close at different speeds, which is why it is hard to see as a single problem.

**Closed vectors.** Every retrieval system converts your documents into numbers using a specific embedding model, then stores those numbers in a specific index. Both halves are traps. The index may be a proprietary store with no meaningful export. Worse, the numbers themselves are meaningless outside the model that produced them: change the embedding model and every vector in the system must be recomputed. Most organisations do not know what re-embedding their corpus costs, or how long it takes, or who would do it. That ignorance is the lock. Chapter 35Chapter 35 · 11 min LockedRetrieval Mechanics: Chunking, Hybrid Search, and RerankingThe engineering layer under every context strategy — chunking, hybrid search, reranking, and how to prove it works. covers the mechanics; the governance point is that your chunking strategy and your embeddings are architecture, not settings.

**Opaque caches.** Everything an agentic system learns at runtime — which retrieval worked, which summary was accepted, which answer a user corrected, what got compacted away — lands in some store. If that store is a vendor's internal cache with no schema you can query, then your organisation's operating memory of its own AI is not an asset you own. It is telemetry you are allowed to view. Chapter 11Chapter 11 · 7 min LockedContext Is a Living Layer, Not a DocumentDefinitions drift, macro conditions shift, agents relearn — treat context like code, with ownership, versioning, and continuous review. makes the case that context is a living layer; a living layer you cannot read is somebody else's living layer.

**Private tool registries.** Agents act through tools, and each tool carries a description, a schema, a permission scope, and an approval policy. That collection is the single most expensive artefact in an agentic programme, because every line of it was argued over by security, legal, and the team that owns the system being touched. If it exists only as configuration inside one runtime, then a migration means re-litigating every one of those arguments with the same stakeholders, who will be less patient the second time.

**Walled agent runtimes.** The orchestration itself: how work is planned, decomposed, retried, escalated, and handed between agents. Vendors express this in their own idiom, and the idiom does not translate. This is where the handoff seams described in Chapter 34Chapter 34 · 7 min LockedDesigning Multi-Agent Systems That Actually WorkWhen one agent is not enough — and how to make many cooperate without chaos. get baked into a shape only one platform can execute.

Here is the discipline that decides whether any of this is recoverable: business meaning and model instruction are two different things, and they must live in two different places.

What happens by default is that they fuse. A team needs an agent to understand that active customer excludes accounts in a grace period, so they write that rule into the prompt. Then they add three examples of edge cases, because the model gets it wrong without them. Then they discover this particular model needs the rule stated before the examples rather than after. The resulting artefact is 40 per cent business policy and 60 per cent accommodation of one model's quirks, and the two are now inseparable.

That is the moment portability dies, and it dies quietly. The prompt is not a portable asset, because two-thirds of it is about a model rather than about your business. When the model changes — and it will, whether you choose it or the vendor deprecates it underneath you — you are not adapting the prompt. You are rediscovering which parts of it were your company's rules.

The fix is structural and unglamorous. Business meaning belongs in a form that has nothing to do with any model: definitions with owners, metrics with formulas, entities with relationships, policies with effective dates. This is the semantic layer of Chapter 7Chapter 7 · 7 min LockedSemantic Layers, Ontologies, and Metrics in Simple WordsThe big context words, explained without the jargon. and the business memory of Chapter 8Chapter 8 · 8 min LockedFrom Data Catalog to Business MemoryCatalogs describe data. Memory describes meaning.. The model-specific layer then becomes a thin, disposable adapter: how to phrase this definition for this model, in this order, with these examples. When the model changes you rewrite the adapter, which is cheap, and you keep the meaning, which is not.

The test is simple. Open the artefact that tells your agent what your business means. If the word you or a phrasing instruction appears in it, meaning and instruction have fused, and you have work to do.

If meaning is separated from instruction, the natural next question is what shape meaning should take so that it survives a model change. Chapter 12Chapter 12 · 7 min LockedPortable Context: The Open Contract for AgentsContext is your IP — it must move across agents, models, and clouds through an open contract, not sit locked inside one runtime. calls this portable context: an explicit contract between your organisation and any model that works for it.

A workable contract has five parts. **Definitions** — the entities, metrics, and policies, each with an owner and a version, in a plain structured format your own systems can read. **Provenance** — for every fact, which system of record settles it, so a new model can be pointed at evidence rather than trained into agreement. **Precedence** — which source wins when two disagree, because ambiguity is where agents fail most expensively and most silently. **Scope** — who may see what, expressed against your identity system rather than a vendor's internal permission model. **Evidence of correctness** — the evaluation cases that define acceptable behaviour, including the failures you have already suffered.

That last part is the one teams skip, and it is the one that makes the rest usable. An evaluation suite is a portability instrument, not just a quality instrument. It is the only thing that lets you answer the question a model migration actually poses: is the new configuration as good as the old one? Without it, every model change is an act of faith, which means in practice that you never make one. Chapter 21Chapter 21 · 6 min LockedQuality, Speed, and Cost TradeoffsHow to balance accuracy, latency, and spend. treats this as a cost and quality discipline; read it also as the thing that keeps your options open.

Note what the contract does not require. It does not require you to leave your platform, avoid proprietary features, or build everything yourself. It requires that the authoritative copy of your meaning lives somewhere you control, in a format you can read, and that the vendor's version is a fast projection of it rather than the original.

One specific correction saves more future pain than any other, so it deserves its own section: treat vector representations as derived artefacts, not as source of truth.

A vector is a lossy numerical summary of a chunk of text, computed by a particular model at a particular time. It is a build output. Yet organisations routinely treat their vector store as a system of record — the only place certain content exists in retrievable form — and then discover they cannot change embedding models without a project.

The correct posture is the one your data team already applies to everything else. The source content lives in open tables, with its chunk boundaries, metadata, ownership, and permissions as columns you can query. Embeddings are a materialised view over that, regenerable by definition. Then changing the embedding model is a recompute with a known cost, not a migration with an unknown one. Two teams can even run competing retrieval configurations over the same corpus and compare them honestly, which is how you find out that a cheaper model is good enough — the argument made in our essay on small models and big context.

This is also what makes retrieval auditable. When a regulator or an internal reviewer asks why the agent cited this document, the answer must be a query against something you own, not a support ticket to a vendor.

None of this argues for building your own platform. Assembling everything yourself is a different failure mode with a worse cost curve. The practical position is a small number of open seams in specific places, with proprietary excellence everywhere else.

**Seam one: tool interfaces.** Define the actions your agents may take against an open protocol rather than a runtime-specific format, so the same tool is callable from a different orchestrator without re-approval. This was the substance of the protocol shift we covered in MCP, A2A, and the year choice stopped being theoretical.

**Seam two: context storage.** The definitions, chunks, provenance, and precedence rules live in your own open tables. The vendor gets a copy; you keep the original.

**Seam three: traces and evaluations.** Every agent run — inputs, retrievals, tool calls, outcome, cost, and human verdict — lands in tables you own, in a schema you chose. This is the single highest-leverage seam, because it simultaneously gives you the P&L discipline of Chapter 18Chapter 18 · 6 min LockedThe Hidden Cost of Agentic AIWhere the dollars actually go. and the evidence base for any future migration.

**Seam four: model access.** Calls go through an internal boundary rather than being scattered through application code, so that changing or adding a model is a configuration change. Not because you plan to switch weekly, but because the ability to switch is what makes the conversation with your vendor a negotiation. Chapter 25Chapter 25 · 6 min LockedPlatform Independent, Platform AwareLearn the concepts. Map them to Databricks, Snowflake, AWS, Azure, GCP, and open source. calls this being platform independent and platform aware; our essay on one model as a single point of failure covers the operational case.

Everywhere else — the fast index, the governed runtime, the managed serving, the good defaults — use the best proprietary thing you can buy. Chapter 24Chapter 24 · 5 min LockedBuild for ChangeDesigning for change you can't predict. frames the principle correctly: build for change at the seams, not everywhere.

Architecture arguments go in circles because both sides are reasoning about hypotheticals. Replace the argument with a question that has a factual answer.

**If your primary AI vendor doubled its price, was acquired, or deprecated the model your agents depend on, what would it take to run the same agents on a different substrate, and how do you know?**

Ask it in a room with the people who would actually do the work, and insist on a number with a unit. The answers sort themselves into four groups. Some teams say two weeks, and can name the components that would change — they are sovereign. Some say a quarter, with a plan — that is a healthy position for most organisations. Some say a year, which is not a plan but is at least honest. And some cannot answer at all, which is the only truly bad outcome, because it means the exposure is unmeasured rather than accepted.

The goal is not to reach two weeks. It is to make the number known and deliberate. An organisation that has decided a quarter is acceptable has made a strategy. An organisation that does not know has made an assumption on behalf of a future leadership team that will not thank it. Our earlier essay on the exit test applies the same logic to contracts; this is the same test pointed at architecture.

One warning about answering it on paper. The cost of leaving is always understated by the people who built the thing, because they hold in their heads the context that is missing from the system. The only honest version of this test is a small rehearsal: take one real agent, run it end to end on a second substrate, and measure what broke. Most organisations discover that the model was the easy part and the tool approvals were the hard part.

Underneath the architecture is a question about what kind of business you are building. Every organisation is now accumulating an intelligence asset: the encoded understanding of how it operates, what its terms mean, which sources are authoritative, how its agents have been corrected. That asset compounds. It is the thing described in Chapter 10Chapter 10 · 9 min LockedInstitutional Memory Is the MoatAs frontier models converge, the durable advantage is the accumulated why — decisions, definition history, and organizational logic. as institutional memory and in Chapter 5Chapter 5 · 7 min LockedContext Is the New Data LayerWhy meaning sits above storage and compute. as the new data layer.

The only open question is whose balance sheet it lands on. If that understanding lives in a form you own, it is capital: it appreciates, it transfers across model generations, it makes each subsequent agent cheaper to build, and it is genuinely hard for a competitor to copy because it is specific to your business. If it lives only inside a vendor's product, it is a subscription feature. You paid to create it, you benefit from it while you pay, and its value to you ends with the contract.

Both can be the right decision. A team of twelve should not build a portable context layer to hedge against a migration it will never do. But a large enterprise placing a decade-scale bet on agentic operations is making a capital allocation decision whether or not it notices, and the default — meaning accumulating quietly inside someone else's product — is the one option nobody would choose on purpose.

You did not win the format war to lose the meaning war. Keep the data open, buy the best intelligence you can, and make sure the understanding stays yours.