Buyer's guide
The AI vendor scorecard.
Twelve questions across Context, Control, Cost, and Choice. Score any AI vendor in five minutes and see the lock-in before it shows up on an invoice.
Why most AI vendor evaluations miss the point
Most evaluations compare demos and price per token. Both change every quarter. What does not change is whether the vendor understands your business, lets you govern what agents do, keeps cost predictable, and lets you leave. Those are the 4 C's, and they decide whether a purchase still looks good in year two.
Score a vendor now
Answer each question honestly. Nothing is saved or sent anywhere.
- Context
Can it use our business definitions, not only raw documents?
Metrics, entities, and policies from a governed layer.
- Context
Does it respect source permissions at answer time?
A user only sees answers built from data they can access.
- Context
Can we inspect what context was used for an answer?
Citations and retrieved sources are visible and logged.
- Control
Can we gate risky actions behind approval?
Writes, payments, and emails pause for a human.
- Control
Is there a full audit trail per agent and per user?
Every tool call, input, and output is recorded.
- Control
Can we revoke an agent's access instantly?
Short-lived credentials and a kill switch.
- Cost
Is pricing predictable at our expected volume?
You can model cost per task, not only per token or seat.
- Cost
Can we set budgets and hard limits?
Per team, per agent, with alerts before overrun.
- Cost
Can it route easy work to cheaper models?
Not every task needs the most expensive model.
- Choice
Can we export our prompts, evals, memory, and logs?
In open formats, without a services contract.
- Choice
Can we swap the underlying model?
More than one model provider is supported.
- Choice
Does it use open interfaces like MCP or open table formats?
Your tools and data are not rewritten for one vendor.
0 of 12 answered
How to read the result
- 28 to 36. Strong fit. Confirm with a pilot on real data and a written exit plan.
- 18 to 27. Workable with conditions. Put the weak answers into the contract.
- Below 18. Expect rework. Fine for a prototype, risky as a platform.
- Any pillar under 6 of 9. Treat it as a red flag regardless of the total.
The questions behind each pillar
Context
A vendor that only reads documents will give fluent answers with the wrong definitions. Ask how it uses your semantic layer and permissions. See Chapter 6: Business Meaning Beats Raw Retrieval.
Control
Agents act, so you need approvals, audit trails, and instant revocation. See Chapter 15: Guardrails, Approvals, and Audit Trails and Chapter 16: Trust Is Designed, Not Assumed.
Cost
The real number is cost per correct task, including retries and review time. See Chapter 19: Not Every Task Needs the Best Model and Chapter 20: Budget-Aware AI Design.
Choice
Your prompts, evals, and memory are the expensive part to rebuild. Make sure you can take them with you. See Chapter 23: Open Formats, Open Interfaces, Open Thinking and Chapter 12: Portable Context.
Five contract clauses worth asking for
- Export of prompts, evaluation sets, memory, and logs in open formats at no extra cost.
- Support for at least two model providers, or a documented model swap path.
- Hard spend limits with alerts, set by you, enforced by them.
- Audit log retention for at least 12 months, accessible through an API.
- Answers respect source-system permissions, with a test you can run.
Frequently asked questions
What is an AI vendor scorecard?
A short, structured set of questions used to compare AI vendors on the things that decide long-term success: how they handle your context, how much control you keep, what they really cost, and how easily you can leave.
What are the 4 C's of AI vendor evaluation?
Context, Control, Cost, and Choice. Context checks whether the vendor understands your business. Control checks governance. Cost checks real unit economics. Choice checks portability and lock-in.
How do I avoid AI vendor lock-in?
Keep prompts, evaluation sets, memory, and logs in formats you own, require model portability in the contract, and run an exit test before you sign: could you move in 90 days?
What score should a vendor reach?
Treat any pillar below 2 out of 3 as a red flag regardless of the total. A strong total with zero on Choice usually means a cheap start and an expensive exit.
Keep reading
Go deeper than a page.
The Context Advantage is the full 36-chapter living book on Context, Control, Cost, and Choice — written for data + AI professionals.