Retrieval-augmented generation (RAG)

Look the answer up first, then let the model write it.

RAG is the standard pattern for making a model answer from your documents rather than its training. When a question arrives, a search step finds the most relevant passages from your own corpus, those passages are placed in front of the model, and the model is instructed to answer from them and to cite them.

Nearly every failure attributed to the model in a RAG system is actually a failure of the search step. If the right passage was not retrieved, no amount of model quality rescues the answer — and the model will usually produce a confident answer anyway, from whatever it was given.

It is also the cheapest way to keep a system current. Update the documents and the answers change, with no retraining involved.

Why it matters here

This is the architecture behind almost every credible public sector use of AI on policy, legislation and case material, and it is the one where citation is achievable. A system that cannot show the passage it relied on should not be used for anything a citizen can appeal.

The question to ask

When the answer is wrong, can we see whether the retrieval or the generation failed?

Reviewed 2026-09-20 · All decoders