RAG vs fine-tuning: which problem are you actually solving?

RAG vs fine-tuning gets framed as two routes to the same destination, and that framing is what causes the wrong choice. Retrieval gives a model access to information it wasn’t trained on. Fine-tuning changes how the model behaves. They address different failures, and picking the wrong one produces a system that’s expensive, hard to update, and still wrong.

The distinction that decides it

Ask what’s failing.

If the model doesn’t know something — your product’s docs, this customer’s history, yesterday’s prices — that’s a knowledge gap, and retrieval fills it. No amount of fine-tuning makes a model reliably recall a fact you can just hand it, and a fine-tuned model still can’t know what happened after training.

If the model knows enough but responds wrong — ignores your format, adopts the wrong tone, over-explains, picks the wrong tool from a set it clearly understands — that’s a behaviour gap, and fine-tuning is the tool built for it.

The trap is that behaviour problems often look like knowledge problems. “It gives bad answers about our product” might be missing docs, or it might be that the docs are retrieved fine and the model won’t follow your answer format. Those need opposite fixes, so diagnose before you build.

What each actually costs

RAG costs infrastructure and latency. You need chunking, embeddings, a vector store, a retrieval step, and re-ranking once naive similarity stops being good enough. Every query pays a retrieval round trip, and every answer pays for the retrieved tokens in context — which is a real, recurring line item.

Its virtue is that updating knowledge means updating documents. Add a page, it’s answerable in minutes. Delete a page, it stops being cited. For anything that changes weekly, this is decisive.

Fine-tuning costs data and iteration. You need hundreds to thousands of good examples — and “good” is the hard part, since data quality dominates quantity here. Then you evaluate, and usually do it again.

Its virtue is at inference: no retrieval hop, and often shorter prompts because the behaviour is baked in rather than instructed. At high volume that’s a genuine cost and latency win.

The asymmetry that matters: RAG is cheap to change and costs more per call. Fine-tuning is expensive to change and costs less per call. Most teams underestimate how often their requirements change.

Side by side

Criterion	RAG	Fine-tuning
Fixes	Missing knowledge	Wrong behaviour
Update cycle	Minutes	Days to weeks
Setup cost	Infrastructure	Labelled data
Per-call cost	Higher (retrieved tokens)	Lower (shorter prompts)
Latency	Retrieval hop added	None added
Citations	Natural — you have the source	Not possible
Freshness	Live	Frozen at training
Access control	Per-document, enforceable	None — baked in

Two rows deserve emphasis.

Citations. RAG can show its source because it had one. A fine-tuned model can’t tell you where an answer came from, which rules it out anywhere provenance is required.

Access control. With retrieval you filter documents by the user’s permissions before they reach the model. With fine-tuning, anything in the training set is available to everyone who can call the model. If your corpus spans permission boundaries, fine-tuning on it is a data leak with extra steps.

Try the cheap thing first

Before either, exhaust prompting. A better system prompt and three good few-shot examples resolve a surprising share of behaviour complaints in an afternoon, at zero infrastructure cost.

The honest ordering:

1. Prompt engineering. Hours. Fixes more than people expect.
2. RAG. Days. Fixes every knowledge gap and stays current.
3. Fine-tuning. Weeks. Fixes behaviour that survived 1 and 2.

Skipping to step 3 because it sounds more serious is the most common expensive mistake in this space.

When you want both

They compose, and for mature systems the combination is often right: fine-tune for the shape of the output, retrieve for the facts that go in it.

A support agent is the standard example. Fine-tuning teaches it your escalation format, your tone, and when to hand off. Retrieval supplies the current documentation and this customer’s ticket history. Neither substitutes for the other, and the split is clean — behaviour is stable, knowledge changes daily.

For agents specifically

If you’re building an agent rather than a Q&A system, the calculus shifts toward retrieval, for two reasons.

Agent knowledge is inherently live — the user’s calendar, the state of the inbox, what a tool just returned. None of that can be trained in. And what belongs in context on a given step is a retrieval problem by nature.

The one place fine-tuning earns its cost in agents is tool selection at scale. When there are hundreds of tools, a model tuned on your specific tool set picks better than a general one reading descriptions — though retrieving a relevant subset per turn usually gets you most of the way for far less effort.

How to choose

Use RAG when the knowledge changes, provenance matters, access control matters, or you’re still learning what the system needs to know.

Use fine-tuning when behaviour is wrong after real prompting effort, volume makes per-call savings material, or you need a consistent output format that instructions keep failing to enforce.

Use both when the format is fixed and the facts aren’t.

The diagnostic question: if I pasted the right document into the prompt, would the answer be correct? If yes, it’s retrieval. If it would still be wrong, it’s behaviour.

See also: routing between models by task and where the per-call cost actually goes.