RAG bolts an information-retrieval step onto a language model so it answers from a chosen set of documents rather than from frozen training weights alone. Introduced in a 2020 paper by Lewis et al., it pairs a parametric model with a non-parametric external memory reached at inference time, and has since become a default pattern in production AI systems.
The retrieve, augment, generate pipeline
The core idea is a three-stage loop that runs on every query. First, the reference corpus is turned into embeddings, numeric vectors in a high-dimensional space, and stored in a vector database. This corpus can be unstructured text, semi-structured data, or structured sources such as knowledge graphs. Second, given a user query, a retriever selects the most relevant documents by comparing vectors, with the comparison method depending on how the index was built. Those documents are then folded into the model’s prompt, a step sometimes called augmentation. Third, the model generates an answer conditioned on both the original query and the retrieved passages.
The mechanism that makes augmentation work has a blunt nickname: prompt stuffing. Without it, the model sees only what the user typed; with it, the retrieved context is injected early in the prompt to push the model to prioritize the supplied facts over its pre-existing training knowledge. IBM frames the final step as the model drawing on both the augmented prompt and its internal representation of training data to synthesize the answer.
Why grounding beats retraining
The strongest argument for RAG is economic and operational. When new information appears, you update the external knowledge base instead of retraining the model, which sidesteps the compute and financial cost of a training run. That decoupling lets a general model draw on domain-specific or freshly updated data it never saw in training, so a chatbot can answer from internal company documents or an authoritative source on demand.
RAG also attacks hallucination, the failure mode behind real-world damage: chatbots inventing company policies, or fabricating legal cases for lawyers hunting citations. As Ars Technica puts it, RAG blends the model with a document look-up process to help it stick to the facts. A side benefit is transparency. Because answers can cite the passages they came from, users can cross-check the sources rather than trusting the model blind.
Retrieval quality is the real bottleneck
RAG improves accuracy but does not cure hallucination, and the reason is instructive: grounding is only as good as what the retriever fetches and how the model reads it. A model can produce misinformation even from factually correct sources if it misreads context. MIT Technology Review’s example is sharp: a system answered that “The United States has had one Muslim president, Barack Hussein Obama,” having retrieved a rhetorical book-chapter title, “Barack Hussein Obama: America’s First Muslim President?”, and taken it literally. The model did not understand the title, so it manufactured a false claim from a true source.
This is the failure class sometimes labeled RAG poisoning. Retrieved sources can be correct but misleading, the model can strip a statement from its context, and when sources conflict the system may not know which to trust. The worst case merges outdated and current information into a single confident but wrong answer. Two related weaknesses compound it: RAG reduces but does not eliminate the need for retraining, and models often fail to recognize when they lack enough information to answer, generating a response where they should signal uncertainty.
The levers: encoding, retrieval, chunking, and search
Because retrieval quality decides output quality, most engineering effort targets the pipeline rather than the model. The levers cluster into a few areas.
On encoding, the choice is between sparse vectors, which are dictionary-length, mostly zero, and encode word identity, and dense vectors, which are compact and encode meaning. Similarity scoring can be sharpened with dot products, retrieval sped up by swapping exact K-nearest-neighbor search for approximate nearest neighbor, and ranking refined after the fact with late-interaction methods such as ColBERT. Hybrid schemes combine dense and sparse representations, and approaches like SPLADE add query expansion to lift recall.
On the retriever itself, techniques include pre-training with the Inverse Cloze Task, supervised optimization that aligns retrieval probabilities with the generator’s likelihoods by minimizing KL divergence, and reranking to push the most relevant documents to the top.
Chunking, the strategy for splitting documents into vectorizable pieces, is a deceptively important lever. Fixed-length chunks with overlap are fast and preserve context across boundaries; syntax-based chunking splits on sentences using tools like spaCy or NLTK; and format-aware chunking respects a file’s natural structure, keeping whole functions or classes together in code and leaving HTML tables or images intact.
Hybrid search addresses a specific gap: pure semantic vector search can miss key facts, so combining it with traditional full-text search and merging both result sets through scoring or reranking catches what either method alone would drop. A more radical option is redesigning the model around the retriever, as in Retro, where a network 25 times smaller reaches comparable perplexity by offloading domain knowledge to retrieval, though it reintroduces the training cost RAG was meant to avoid.
Key takeaways
- RAG runs retrieve, augment, generate on every query: embed a corpus, fetch relevant passages, stuff them into the prompt, then generate a grounded answer.
- Its main win is updating a knowledge base instead of retraining the model, plus citable sources for transparency.
- It reduces hallucination but does not eliminate it; a model can still produce falsehoods from correct-but-misread sources.
- Output quality is bounded by retrieval quality, so the practical work lives in embeddings, retrieval strategy, chunking, and hybrid search.
- Chunking strategy and hybrid (semantic plus full-text) search are high-leverage and easy to underrate.
Sources
- Read the source: en.wikipedia.org/wiki/Retrieval-augmented_generation
- Full text: retrieval-augmented-generation
- Related: i-built-an-llm-from-scratch