The Problem RAG Solves
A large language model’s knowledge is frozen at training time and bounded by what appeared in its training data. Ask it about something that happened after its cutoff, or about a private document it never saw (your company’s internal wiki, a customer’s account history, a codebase), and it has two options: say it doesn’t know, or guess — and LLMs guess fluently, which is what makes hallucination dangerous rather than obviously wrong. Retrieval-Augmented Generation (RAG) addresses this by giving the model a lookup step before it answers: search an external knowledge source for relevant material, insert that material into the prompt, and have the model generate its answer grounded in what was actually retrieved rather than purely from memorized parameters.
The term and the core architecture come from Patrick Lewis and colleagues’ 2020 paper, which combined a pretrained sequence-to-sequence generator with a dense passage retriever indexed over Wikipedia, and showed it beat purely parametric models on knowledge-intensive question answering while producing more specific, factual, and diverse outputs (Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020). The paper’s own framing is useful: rather than trying to cram all of a system’s knowledge into a model’s weights, keep the knowledge in an external, updatable store and give the model a way to consult it — closer to an open-book exam than requiring the model to memorize the textbook.
Core Architecture
A production RAG system has two halves working together:
The retriever takes a user’s query, converts it into a vector embedding (a dense numerical representation where semantically similar text ends up close together in vector space), and searches a vector database for the stored document chunks whose embeddings are closest to the query’s embedding. This is semantic search, not keyword search — it can retrieve a passage about “canine companions” for a query about “dogs” even with zero shared words, because the embeddings capture meaning rather than exact tokens.
The generator (the LLM itself) receives the original query plus the retrieved passages, stitched into a single prompt, and produces an answer that’s expected to be grounded in that retrieved context rather than free-associated from training data alone.
The full pipeline, end to end: a source document gets split into chunks → each chunk is embedded into a vector → vectors are stored in a vector database, indexed for fast similarity search → at query time, the query is embedded the same way → the database returns the top-k most similar chunks → those chunks plus the query go into the model’s context window → the model generates its answer.
RAG vs. Fine-Tuning
Both are ways to make a general-purpose model behave well on a specific domain, but they solve different problems and are frequently confused:
| RAG | Fine-Tuning | |
|---|---|---|
| What it changes | Nothing in the model — adds external context at query time | The model’s own weights |
| Best for | Injecting current facts, private/proprietary data, source citations | Teaching a new skill, tone, or output format |
| Update cost | Cheap — re-index the data, no retraining | Expensive — requires a full or partial retraining run |
| Data freshness | As current as the last index update | Frozen again at the next training cutoff |
| Traceability | Can cite which retrieved passage supports an answer | No way to point to “why” the model said something |
| Failure mode | Bad retrieval → irrelevant or missing context | Overfitting, catastrophic forgetting of prior skills |
The two aren’t mutually exclusive — a fine-tuned model can still sit behind a RAG pipeline, and often does in production systems that need both a specific behavior style and access to fresh external data.
The Practical Components
Chunking strategy: source documents rarely map cleanly onto “one chunk per idea.” Chunk too large and irrelevant surrounding text dilutes the embedding’s specificity; chunk too small and you lose surrounding context a passage needs to make sense on its own. Most production systems chunk by a fixed token count with overlap between consecutive chunks, or split along natural document boundaries (headings, paragraphs) where the source format allows it.
Embedding models: the model that converts text into vectors is a separate choice from the generator LLM, and it directly determines retrieval quality — a weak embedding model will return semantically-off passages no matter how good the generator is downstream.
Vector databases: the storage and similarity-search layer. The field has settled into a few clear categories rather than one winner: pgvector (a Postgres extension) is the natural choice for teams already running Postgres and wanting to avoid adding a new datastore to their infrastructure; managed, zero-ops services like Pinecone remove index-tuning and scaling work entirely; Weaviate bundles built-in hybrid search (combining traditional keyword/BM25 search with vector similarity in one query) and can auto-generate embeddings on insert; Chroma is optimized for fast local prototyping with minimal setup. Each has an official documentation site worth checking directly for current capabilities rather than relying on comparison roundups, since this space moves quickly: pgvector, Pinecone, Weaviate, Chroma.
Advanced Retrieval Patterns
Naive “embed the query, grab the top-k chunks” retrieval leaves obvious quality on the table, and several now-standard techniques address specific weaknesses in it:
Re-ranking: run a cheap, broad retrieval pass to pull a larger candidate set (say, the top 50 chunks) than you’ll actually use, then apply a slower but more accurate model to re-score and re-order just those candidates before keeping only the true top-k. This two-stage approach trades a small amount of latency for meaningfully better precision, since the expensive re-ranker only has to evaluate 50 candidates instead of the entire corpus.
Hypothetical Document Embeddings (HyDE): rather than embedding the user’s raw query, first ask an LLM to generate a hypothetical answer to it, then embed that generated text and use it for the similarity search. The original paper’s finding is counterintuitive but well-supported: even when the generated hypothetical document contains factual errors, its semantic structure and phrasing tend to resemble real relevant documents far more closely than a short, sparse user query does, which improves retrieval without needing any labeled relevance data (Gao et al., Precise Zero-Shot Dense Retrieval without Relevance Labels, 2022).
Hybrid search: combine traditional keyword/BM25 search with vector similarity in the same query, rather than relying on embeddings alone. Keyword search still wins on exact-match cases embeddings can miss entirely — a product SKU, an error code, a person’s name — while vector search wins on paraphrased or conceptual queries; hybrid retrieval takes both signals rather than betting the whole system on one.
Evaluating a RAG System
“It feels like it’s working” isn’t a metric, and RAG systems fail in two genuinely separate places that need separate evaluation: retrieval quality (did the system find the right passages at all) and generation quality (given good passages, did the model use them correctly). A retrieval-quality failure and a generation-quality failure look identical from the outside — a wrong answer — but require completely different fixes, so conflating them wastes debugging effort. Practical evaluation typically checks retrieval separately (does the top-k set actually contain the passage that answers the question, measured against a held-out set of question/answer pairs with known source passages) and generation separately (given the correct passages, does the model’s answer actually reflect them, and does it avoid asserting anything the passages don’t support). Tooling in this space (frameworks like RAGAS) has emerged specifically to automate this two-part evaluation rather than relying on end-to-end “does the final answer look right” spot-checks, which tend to miss exactly this class of bug.
Real Limitations
RAG doesn’t eliminate hallucination, it narrows where it can come from. The system is only as good as its retrieval step: if the retriever returns irrelevant or incomplete passages, the generator either hallucinates around the gap or answers confidently from an incomplete picture, and a plausible-sounding wrong answer grounded in the wrong retrieved passage is arguably harder to catch than an obviously made-up one. Context window size also caps how many retrieved passages can actually be included, which is a real tradeoff against retrieving broadly enough to cover a genuinely ambiguous query. And RAG adds real infrastructure and latency — an embedding call and a database round-trip before the generation step even starts — that a purely parametric model doesn’t need.
This is also the retrieval half of the loop that agentic AI systems use when a “tool” is a knowledge lookup rather than an external action — many production agents treat their RAG pipeline as just one more tool the model can decide to call.
