RAG vs fine-tuning.

The short answer

Use retrieval when the problem is knowledge: the model does not know your policy, your customer, your latest price list, or what happened yesterday. Retrieval is how facts arrive at inference time, which means they can be updated in minutes and cited back to a source.

Use fine-tuning when the problem is behaviour: the model knows enough but answers in the wrong format, the wrong tone, the wrong reasoning shape, or with too many tokens for your latency budget. Fine-tuning moves the default, permanently, and cannot be cited.

If your failure log says wrong fact, reach for retrieval. If it says right fact, unusable answer, reach for fine-tuning.

Side by side

• What it changes. RAG: the information in the request. Fine-tuning: the weights.
• Freshness. RAG: as fresh as your index. Fine-tuning: frozen at training time.
• Attribution. RAG: every claim can point at a source. Fine-tuning: no provenance.
• Unit cost. RAG: higher per call, because you pay for retrieved tokens on every request. Fine-tuning: high one-off cost, then cheaper and shorter prompts.
• Latency. RAG: adds a retrieval hop plus reranking. Fine-tuning: adds nothing at inference and often removes tokens.
• Governance. RAG: permissions can be enforced at query time. Fine-tuning: anything in the training set is baked in, including data you later need to delete.
• Iteration speed. RAG: change a document, behaviour changes. Fine-tuning: change a behaviour, run a job and re-evaluate.
• Portability. RAG: the index and the context contract move with you. Fine-tuning: the artifact is tied to a base model, which is a lock-in surface.

The crossover point

There is a real economic crossover. Retrieval costs scale with volume, because every request carries retrieved tokens. Fine-tuning costs are mostly fixed, then amortise. At low volume with changing knowledge, retrieval wins on every axis. At high volume with stable behaviour and a repeated task shape, a small fine-tuned model can beat a large general one on cost and latency at equal accuracy — sometimes by an order of magnitude.

The practical test: compute cost per correct answer for both, not cost per call. A cheaper call that is wrong more often is not cheaper. The second test is the freshness question — if your knowledge changes weekly, no amount of training will keep up, and you are choosing retrieval whether you like it or not.

What most production systems actually do

They fine-tune, or select, a small model for the shape of the task and keep retrieval for the facts. The fine-tune handles output format, refusal behaviour, tool-call syntax, and domain vocabulary. Retrieval handles the sentence that will be wrong tomorrow. Evaluation sits over both, because the moment you change either, you need a regression gate to tell you which direction quality moved.

Before you decide

• Write down ten real failures and label each one knowledge or behaviour.
• Measure your current cost per correct answer, with a golden set, before changing anything.
• Check whether the knowledge involved has a retention or deletion obligation. If it does, do not train on it.
• Ask what happens when the base model is deprecated. Retrieval survives that; a fine-tune does not.