Retrieval-Augmented Generation

WhenWhen a language model answers purely from what it absorbed during training, it is doing something closer to reciting from memory than looking anything up — and, as the previous page on question-answering systems covered, that memory is measurably worse the rarer or more recent the fact is. Retrieval-Augmented Generation, universally shortened to RAG, is the standard fix: instead of asking the model to answer from memory alone, the system first searches a body of text for passages relevant to the question, then hands those passages to the model alongside the question and asks it to answer using them. Lewis and colleagues, who introduced the term and the architecture together, framed it explicitly as combining a model's learned, "parametric" knowledge with an external, non-parametric memory that can be inspected, updated, and swapped out without retraining anything [1] — which is also exactly the mechanism at work whenever a student uploads a document to a web-based AI tool and starts asking questions about it.

Why RAG Exists

RAGRAG solves two separate problems that closed-book generation cannot, on its own, solve at all. First, a model's parametric knowledge is frozen at training time and skewed toward whatever was well-represented in its training data — it has no reliable way to answer questions about a document that did not exist when it was trained, or a fact that was rare enough in its training data to be poorly learned in the first place [2]. Second, and just as importantly, grounding an answer in a specific, inspectable passage gives both the system and the user something to check the answer against — an open-book answer can be verified by reading the cited passage; a closed-book answer can only be verified by separately researching the claim from scratch, which almost never actually happens in ordinary use. Neither problem is really about the model being unintelligent; both are structural consequences of where the knowledge is stored and whether that storage location can be examined.

The Basic Pipeline

Conceptually,Conceptually, and without needing to write a line of code to understand it, a RAG system does four things before it ever generates a word of the answer. It indexes a collection of documents by splitting them into smaller passages ("chunks") small enough to be individually relevant and large enough to still make sense on their own. It converts each chunk into an embedding — a numeric representation positioned so that passages with similar meaning end up numerically close together, regardless of whether they use the same words. When a question arrives, it embeds the question the same way and retrieves the chunks whose embeddings are closest to it. Finally it augments the question with those retrieved chunks and asks the model to generate an answer grounded in them. This is precisely what is happening, invisibly, the moment a web-based AI tool lets a user attach a PDF and start asking questions about it: the upload triggers indexing, and every question you type triggers a fresh retrieve-and-generate cycle behind the scenes.

The single biggest practical consequence of this design is that retrieval quality is a ceiling on answer quality: if the right chunk is never retrieved — because it was split awkwardly across a chunk boundary, because the wording of the question doesn't embed close to the wording of the passage, or because it simply wasn't in the indexed collection — no amount of generation skill downstream can recover the missing information, and the model will typically fall back on its own parametric guess without flagging that it has done so.

Multi-Step and Reasoning-Aware Retrieval

TheThe basic pipeline above runs retrieval exactly once per question, which works well for questions a single passage can answer outright and works badly for questions that require combining facts scattered across several different passages. Yang and colleagues built the HotpotQA benchmark specifically to expose this failure mode: questions deliberately constructed so that no single retrieved passage contains the full answer, forcing a system to find one supporting fact, then use it to work out what to look for next [3]. A question like "which of two directors, born in the same country as a particular actor, directed the more recent film" cannot be answered from one retrieval pass — the country has to be established first, before the second, dependent search even has a query to run.

Two related lines of work address this by making retrieval itself iterative and reasoning-driven rather than a fixed one-shot lookup. Trivedi and colleagues' IRCoT interleaves retrieval with step-by-step chain-of-thought reasoning: the model generates one reasoning step, that step's content is used to retrieve the next batch of passages, those passages inform the next reasoning step, and so on — reasoning steers what gets searched for next, and what gets retrieved steers what the model can reason about next, in a loop rather than a single pass [4]. Asai and colleagues' Self-RAG goes a step further by training the model to decide, adaptively, when it needs to retrieve at all — issuing special tokens that trigger a fresh search only when the model judges its current information insufficient, and separately reflecting on whether a retrieved passage actually supports the claim it's about to make [5]. Both are what "reasoning-aware retrieval" means in practice: the search query on step two is not fixed in advance, it is generated by reasoning about what step one's result was still missing.

RAG's Real Limitations

NoneNone of this makes RAG a solved problem, and treating it as one is itself a common and consequential mistake. Barnett and colleagues, reporting on real RAG deployments across research, education, and biomedical case studies, catalogue seven recurring failure points spanning the entire pipeline — content that should have been retrieved but wasn't because it fell outside the top-ranked chunks, chunks retrieved but never actually included in what got sent to the model because of context-window limits, and answers extracted from the right context but still assembled incorrectly [6]. Chunking is a particularly sharp source of this: a passage split at the wrong boundary can sever a fact from the sentence that qualifies or contradicts it, so the retrieved chunk looks relevant while quietly missing the detail that would have changed the answer. And critically, RAG reduces hallucination without eliminating it — Ji and colleagues' survey of hallucination in language generation is explicit that supplying a source narrows a model's opportunities to fabricate but does not remove its tendency to blend genuine source content with unsupported, fluently-stated additions [7]. A wrong answer produced with a document attached is not automatically more trustworthy than one produced without — it is only more trustworthy once someone has actually checked it against the passage the system claims to be citing.

References

1. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems, 33, 9459–9474.
2. Roberts, A., Raffel, C., & Shazeer, N. (2020). How Much Knowledge Can You Pack Into the Parameters of a Language Model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 5418–5426. https://doi.org/10.18653/v1/2020.emnlp-main.437
3. Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., & Manning, C. D. (2018). HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2369–2380.
4. Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2023). Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL).
5. Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2024). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. In The Twelfth International Conference on Learning Representations (ICLR).
6. Barnett, S., Kurniawan, S., Thudumu, S., Brannelly, Z., & Abdelrazek, M. (2024). Seven Failure Points When Engineering a Retrieval Augmented Generation System. In Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering — Software Engineering for AI (CAIN), 194–199. https://doi.org/10.1145/3644815.3644945
7. Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12), Article 248. https://doi.org/10.1145/3571730