How RAG actually works

Your RAG demo answered beautifully in the meeting. In production it confidently cited a document from three quarters ago. Here is what is really happening under the hood.

The Felesh team 9 Jul 2026 Updated 10 Oct 2026 9 min

The demo was flawless. You fed it your product documentation, asked a question a real user would ask, and the answer came back precise and fluent. Then you shipped it. A week later a support ticket arrived: the model had confidently cited a pricing tier you discontinued three months ago. The retrieval had pulled the most plausibly similar document — not the most correct one.

That gap, between plausible and correct, is where RAG lives. Understanding it properly requires looking at what RAG actually does, step by step, and why naive implementations break in predictable ways.

Retrieval ranks by similarity, not by date. The retired March page scores 0.91 against the current page's 0.87, so the bot quotes the old price.

What you are choosing between

Before the pipeline, the alternatives. Retrieval-augmented generation (Lewis et al., 2020) is not the only way to give a model access to a corpus.

Long-context stuffing drops everything — or a large slice of everything — directly into the model’s context. It is the simplest architecture: no retrieval infrastructure, no chunking decisions, no index to maintain. It works well when the corpus is small and stable. It fails at scale: token cost rises linearly with corpus size, and attention dilutes as context grows longer. A model asked to synthesise a 200-page document whose answer lives on page 7 can miss it.

Keyword search / BM25 is the classic information-retrieval approach: match tokens, rank by term frequency and inverse document frequency. It is fast, interpretable, and surprisingly hard to beat on exact-term lookups. If your users search for invoice number INV-20240312-007, BM25 will find it. A dense embedding will blur that string into a region of meaning-space where it is surrounded by other invoice documents — close, but perhaps not the right one.

GraphRAG (a design popularised by Microsoft Research) stores the corpus as a knowledge graph and traverses edges at query time instead of doing vector lookup. It handles multi-hop questions well: “who approved the contract signed by the person who reported to the VP of sales in Q3?” can be answered by following edges. It adds significant infrastructure and graph maintenance overhead.

Naive vector (single-vector) RAG is the simplest dense approach: chunk the corpus, embed the chunks, retrieve the top-k by cosine similarity, drop them in context. This is what most tutorials show. It is where most production failures begin.

The real pipeline

The production-grade approach — the one that addresses the most common failure modes — looks like this.

Chunk. Split your corpus into pieces the model can use. Chunk size is a real decision: too small and you lose context; too large and you retrieve a lot of noise around the answer. Overlapping windows help. Metadata tagging at chunk time (document date, section, source) pays dividends later.

Embed. Run each chunk through an embedding model to produce a dense vector. The embedding model is not a footnote: different models have different strengths across domains. A general embedding model trained on web text may underperform on legal or clinical corpora.

Chunk windows overlap, so no sentence is cut in half, and each chunk carries its date, section and source before it becomes a vector.

Retrieve — dense + BM25 in parallel. At query time, embed the query and retrieve top-k by cosine similarity (dense retrieval). Simultaneously, run BM25 over the same corpus. Dense retrieval finds semantically related chunks even when the user’s words differ from the document’s words. BM25 finds exact tokens — model names, IDs, proper nouns — that dense retrieval blurs.

Rerank. Merge the two candidate lists and pass them to a cross-encoder reranker. A cross-encoder sees the query and a candidate together and scores their relevance jointly — it is slower than embedding similarity but significantly more accurate. The reranker does the triage that neither retrieval pass can do alone.

The cross-encoder scores each candidate together with the question. The invoice line only BM25 found ranks first, and the dense near-miss about emailing falls out.

Ground. Drop the top reranked chunks into the model’s context with an instruction to answer from what is provided and to say when it cannot. Grounding instructions matter: without them, the model will mix retrieved content with its parametric knowledge, and the seam is invisible.

Grounded: each answer line cites the chunk it came from, and the question no chunk answers gets “Not in the documents” instead of the model's guess from memory.

This is the architecture production deployments have converged on — a step beyond the dense-only retrieval of the original RAG paper (Lewis et al., 2020). Hybrid retrieval plus reranking addresses the failure modes that pure vector RAG cannot.

Knowledge is not memory

One distinction worth stating plainly, because conflating them causes real design errors.

Knowledge is facts in a corpus — documents, records, policies, code. RAG retrieves knowledge: at query time it fetches relevant facts from a store and passes them to the model as context. The store does not grow because of the conversation; it grows because someone updates the corpus.

Memory is the running state of a relationship — what was said in prior turns, what a user prefers, what happened in a past session. Memory is tracked across time; knowledge is indexed across documents.

They need different stores. A knowledge base is a retrieval index (vector store, search index, graph). Memory is a session log, a preference record, a summary of past interactions. Routing a memory question to the knowledge retrieval pipeline — or vice versa — is a category error. For a deeper look at the memory side, see how agents remember.

Where even good RAG breaks

Hybrid retrieval with reranking solves the most common failures. It does not solve all of them.

Multi-hop questions. “What did the Q2 report say about the supplier mentioned in the renewal clause of the contract the CFO signed last year?” requires chaining: find the contract → find the clause → find the supplier → find the Q2 report → retrieve the relevant passage. Single-pass retrieval, dense or BM25, retrieves one set of chunks. It cannot follow a chain of reasoning across documents. GraphRAG was designed for exactly this; it is worth the infrastructure cost when multi-hop is the dominant query pattern.

Exact-term and ID lookups. Dense embeddings are trained to represent meaning. A product SKU or invoice ID is a token sequence with no semantic neighbourhood worth exploiting. BM25 is better here — but even BM25 can fail if the token is rare or if the index was built with aggressive normalisation. For high-precision ID lookups, a dedicated structured lookup (a database query, not a vector search) is more reliable.

Freshness. A retrieval index is only as current as its last update. If your corpus is updated at midnight and a user asks at 11 pm about an event that happened at 6 pm, the index does not have it. This is not a retrieval failure; it is an architecture failure. Streaming ingestion, incremental indexing, and cache invalidation are operational concerns that RAG does not solve by itself.

The 23:00 question searches an index built at midnight, so the 18:00 price change does not exist for it until the next rebuild.

Attention dilution at top-k. Passing the top 20 retrieved chunks into context does not guarantee the model uses all of them equally. Research community findings (reported, not Felesh results) show models attend more strongly to content at the beginning and end of context than to the middle — sometimes called the “lost in the middle” effect. A small, high-quality top-k is usually better than a large noisy one.

The choice, and where it loses

The principled default for a domain knowledge retrieval system is hybrid dense + BM25 with a rerank stage. It handles the widest range of query types, is robust to vocabulary mismatch, and does not require graph infrastructure or a very large context window.

Where it loses: it is not simple. You are running two retrieval passes instead of one, maintaining both an embedding index and a BM25 index, and adding a reranker in the serving path. Latency increases. Operational surface grows. If your corpus is small, stable, and fits in context, long-context stuffing with a single call is faster to build and cheaper to run. If your users ask mostly exact-term questions against a well-structured corpus, a search index plus BM25 alone may be enough.

The architecture that answered beautifully in the demo was probably naive vector RAG. The one that holds in production is usually not.

For the adaptation decision — when RAG is the right tool versus fine-tuning or prompt engineering — see fine-tune, RAG, or prompt. For memory architecture — the other store, the one RAG does not touch — see how agents remember.