What is retrieval-augmented generation (RAG)?

Retrieval-augmented generation is the technique of giving a language model relevant documents at question time so it can answer from information it was never trained on. Instead of hoping the fact lives in the model’s weights, you fetch it and put it in the prompt. That one move fixes the two problems that make raw models unusable for real work: they don’t know your data, and they confidently make things up.

The problem it solves

A trained model knows what was in its training set, up to a cutoff date, with no notion of your private documents or what changed yesterday.

Ask it about your company’s refund policy and one of two things happens: it says it doesn’t know, or — worse — it invents a plausible policy. The second is the dangerous one, because an invented answer looks exactly like a real one.

RAG removes the guesswork. Retrieve the actual refund policy, put it in the prompt, and the model answers from the text in front of it rather than from memory. The question shifts from “did the model memorise this?” to “did we retrieve the right document?” — which is a problem you can measure and fix.

How the pipeline works

Five stages, two phases.

Ahead of time (indexing):

1. Chunk. Split documents into passages — a few paragraphs each. Too large wastes context and dilutes relevance; too small loses the surrounding meaning. Chunking quality quietly determines everything downstream.
2. Embed. Convert each chunk into a vector that captures its meaning, and store it.

At question time (retrieval):

3. Search. Embed the question the same way and find the chunks whose vectors are closest — semantic similarity, so “how do I get my money back” matches a passage titled “Refunds” without sharing a keyword.
4. Augment. Paste the top chunks into the prompt alongside the question.
5. Generate. The model answers using the retrieved text.

The model never learns anything permanently. Each answer is assembled fresh from whatever retrieval surfaced, which is exactly why updating a document updates the answers within minutes.

Why it beats the alternatives

Two other routes exist to make a model answer about your data, and RAG wins on the axes that usually matter.

Versus a bigger context window — pasting everything into the prompt. Works until “everything” exceeds the window or the cost, and models get less accurate as context grows, burying the relevant passage among irrelevant ones. RAG sends only what’s relevant.

Versus fine-tuning — training the knowledge into the model. Fine-tuning changes behaviour well and adds knowledge badly: it’s slow to update, can’t cite sources, and bakes every document into one model that everyone can query regardless of permissions. RAG updates in minutes, cites naturally, and filters by access before retrieval.

That last point matters more than it sounds. Because RAG fetches documents, you can filter them by the asking user’s permissions first. Fine-tuning can’t — anything in the training data is available to anyone who can call the model.

Where it breaks

RAG moves the failure from generation to retrieval, and retrieval has its own failure modes.

Retrieval misses. If search returns the wrong chunks, the model answers from wrong context — fluently and wrongly. Garbage in, confident garbage out. This is the dominant failure, and naive similarity search hits it often enough that production systems add re-ranking on top.

Chunking splits the answer. When the fact spans a boundary, no single chunk contains it and retrieval never assembles the whole. Overlapping chunks help; they don’t fully solve it.

Stale index. Documents change, embeddings don’t re-compute automatically, and you serve yesterday’s version confidently. Re-indexing on change is a pipeline you have to build and run.

No source is enough. When nothing relevant exists, a good system says so. A bad one retrieves the closest-but-wrong chunk and answers from it anyway. Handling “I don’t have that” is a design decision, not a default.

Why agents rely on it

For agents, retrieval isn’t an add-on — it’s structural, and it shows up in more places than the classic document-Q&A case.

An agent’s most important knowledge is live: the current state of your inbox, calendar and open tickets. None of that can be trained in; it has to be fetched at the moment of the request, which is retrieval by another name. And deciding what belongs in context on a given step — which past messages, which facts — is itself a retrieval problem. Even choosing the right tool from hundreds is retrieval over tool descriptions.

So an agent is running retrieval constantly, over documents, over memory, over tools. RAG is the general pattern; the document case is just its most visible instance.

In one line

RAG fetches relevant information at question time so the model answers from real data instead of memory or invention. It’s how you point a general model at your specific world — and the reason a well-built agent can tell you what’s in your inbox right now, not what was true at its training cutoff.

See also: RAG vs fine-tuning, decided and what an AI operating system is.