💬 Retrieval-Augmented Generation

What is RAG?

Retrieval-Augmented Generation (RAG) is a way to make a language model answer using knowledge it was never trained on. Instead of relying only on what the model memorized during training (its parametric knowledge), RAG first looks up relevant text from an external source — your company docs, a product manual, a wiki — and hands that text to the model as context before it answers.

A useful analogy: a closed-book exam versus an open-book exam. A plain LLM takes the exam from memory alone. RAG lets it open the textbook to the right page first, then answer. The model still writes the answer in its own words, but now it's grounded in a source you control.

Why RAG?

Three problems RAG solves, all of which come up constantly in real systems:

• Hallucination. On its own, an LLM will confidently state things that are plausible but wrong. Giving it the actual source text to answer from dramatically reduces this — it has something real to lean on instead of guessing.
• Stale and private knowledge. A model's training data is frozen at some cutoff date and never included your internal documents. RAG lets you answer questions about last week's release notes or a private contract without retraining the model.
• Provenance. Because you know which documents were retrieved, you can cite them. The user can check the source — impossible with a raw LLM answer.

The alternative to RAG is fine-tuning — baking new knowledge into the model's weights. Fine-tuning is expensive, slow to update, and hard to cite. RAG is cheaper, updates the moment you add a document, and shows its work. For knowledge that changes, RAG almost always wins.

The RAG Pipeline

A RAG system has two phases: an offline phase that prepares your documents once, and an online phase that runs every time a user asks a question.

1. Indexing (offline)
You can't search raw documents efficiently, so first you prepare them. Each document is split into smaller passages called chunks, and each chunk is passed through an embedding model that turns text into a list of numbers (a vector) capturing its meaning. Chunks with similar meaning end up with similar vectors. These vectors are stored in a vector database (e.g. Pinecone, Weaviate, Qdrant, or Postgres with pgvector) built to find nearest vectors fast.

Why chunking matters: you retrieve whole chunks, so chunk size decides how much context each hit carries. Split too small and a chunk is a fragment with no surrounding context; split too large and a single chunk mixes several topics, so a match on one sentence drags in a lot of irrelevant text. Common strategies: fixed-size (e.g. 512 tokens with a little overlap so ideas at boundaries aren't lost), semantic chunking (split on paragraph/section boundaries), and parent-child (search over small chunks but feed the model the larger parent passage).

2. Retrieval (online)
When a user asks a question, you embed the question with the same embedding model, then ask the vector database for the top-k chunks whose vectors are closest to it — "closest" measured by cosine similarity or dot product. Because scanning every vector exactly is slow at scale, databases use approximate nearest neighbor (ANN) search to get almost-perfect results very fast. Top-k is a dial: a small k keeps the context tight but risks missing the right passage; a large k is safer but stuffs the prompt with more noise (and costs more tokens).

3. Augmentation
The retrieved chunks are pasted into the prompt as context, usually with an instruction like "Answer using only the context below; if it doesn't contain the answer, say you don't know." That last clause is what lets a good RAG system refuse instead of inventing an answer.

4. Generation
Finally the model reads the question plus the retrieved context and writes the answer. If retrieval did its job, the answer is grounded in real, citable text.

Advanced RAG Techniques

Basic retrieve-then-generate gets you surprisingly far, but a few techniques handle the cases where it breaks down.

Reranking. Initial retrieval is fast but coarse: it scores the query and each chunk separately and compares vectors. A reranker is a slower, sharper second pass — a cross-encoder that reads the query and a chunk together and judges how well they actually match. You retrieve, say, the top 20 cheaply, then rerank to find the true best 3. It noticeably improves precision, but since it scores every candidate individually, it adds latency.

Hybrid search. Dense (embedding) search matches meaning, so it handles paraphrases well — but it can miss exact strings like an acronym, a product code, or an error ID, because those don't have a rich "meaning" to embed. Sparse search (BM25, classic keyword matching) is the opposite: great at exact terms, blind to paraphrase. Hybrid search runs both and merges the results, getting the strengths of each. This is why "What is RRF?" can fail in pure semantic search but succeed in hybrid.

The catch when merging is that dense and sparse produce different kinds of scores — a cosine similarity and a BM25 score aren't on the same scale, so you can't just add them. Reciprocal Rank Fusion (RRF) sidesteps this by ignoring the raw scores and using only each result's rank (its position in the list). Each document gets a point of 1 / (k + rank) from every list it appears in — where rank is 1 for the top hit, 2 for the next, and so on, and k is a small constant (commonly 60) that softens the gap between the top spots. Add up a document's points across both lists and sort by the total. The effect: a document that ranks decently in both dense and sparse retrieval beats one that ranks #1 in a single list — so results the two methods agree on rise to the top. It's simple, needs no score tuning, and works surprisingly well.

Query transformation. Sometimes the user's wording retrieves badly. Techniques like HyDE (generate a hypothetical answer first, then search with that), query decomposition (break a complex question into sub-questions), and step-back prompting reshape the query before retrieval.

Multi-hop RAG. Some questions need facts from several documents chained together. Multi-hop retrieves iteratively — using what the first retrieval found to inform a second search.

To feel all of this directly, open the interactive RAG playground: pick a question, then move chunk size, top-k, the similarity threshold, the reranker, and the retrieval mode, and watch precision, recall, latency, and the final answer change. Try the "What is RRF?" question in dense mode and see it refuse — then switch to hybrid and watch the answer appear.

Evaluation

You can't improve what you don't measure, and RAG has two stages that can each fail independently — so you evaluate both:

• Retrieval quality: did the right chunks come back? Measured with hit rate (was a relevant chunk retrieved at all?), MRR (how high did it rank?), and NDCG (ranking quality across all hits). If retrieval is bad, nothing downstream can save the answer.
• Generation quality: given the retrieved context, was the answer good? Faithfulness asks whether the answer actually follows from the context (or drifted into hallucination); relevance and completeness ask whether it addressed the question.

Tooling like RAGAS, TruLens, and DeepEval automates much of this. A practical habit: when a RAG answer is wrong, first check which stage failed — a retrieval miss and a generation slip need completely different fixes.