RAG: Grounding Models in Customer Data

⏱ ~7 min read · 🎧 6 min listen · ✎ 3 quizzes · 🧪 ~40 min lab

Every customer engagement reaches this moment: "Great demo — but it needs to answer from our documents. Our policies, our product manuals, our tickets." The model has never seen any of that, and Lesson 01 told you what it does when asked about facts it doesn't have: it confidently makes them up. This lesson is the standard cure — retrieval-augmented generation — and it's the highest-demand pattern in all of customer AI work.

The problem: the facts aren't in the weights

The model's parameters were frozen at training time. Your customer's refund policy, last week's outage postmortem, the v11.3 config format — none of it is in there, and no amount of prompting can extract what was never stored. Worse, the mechanism doesn't say "I don't know" by default: it samples plausible tokens, which for a policy question means a beautifully formatted, fictional policy.

There's exactly one door into the model's head: the context window. If the true policy text is in the prompt, the most plausible continuation is now grounded in it. But Lesson 02 priced the naive version for you: pasting the entire document base into every call is somewhere between expensive and impossible. So the real question is selection — how do you put just the relevant few pages in?

RAG: retrieve, then generate

RAG is a two-step pipeline your code runs on every question. Retrieve: search the customer's document base for the passages most relevant to the question. Generate: put those passages in the prompt — fenced, Lesson 03 style — and instruct the model to answer only from them, citing which passage supports each claim.¹ Grounding in, hallucination out, receipts included.

The retrieval half has its own machinery. Documents get split into chunks (a few hundred tokens each — retrieval returns pieces, not books). Each chunk gets an embedding: a vector that acts as a fingerprint of its meaning, so "can I get my money back?" finds the refund policy even though it shares no keywords with it. A vector database stores the fingerprints and finds the nearest ones to your question's fingerprint in milliseconds.²

Why customers love the result: answers come from their documents with citations a human can check; updating knowledge means updating an index (minutes), not retraining a model; and access control stays enforceable — retrieve only what this user may see. That last one quietly wins security reviews that pure-model solutions fail.

Where RAG breaks — and the FDE's repair ladder

Here's the uncomfortable truth from the field: most RAG failures are retrieval failures. If the right chunk isn't retrieved, the model answers from wrong or missing context — fluently. The generation step can't fix what search didn't find. So when a customer says "the AI answered wrong," your first question is never "which model?" — it's "what did retrieval return?"

The repair ladder, cheapest rung first. Hybrid search: pair embeddings with classic keyword search (BM25), because exact identifiers like "error TS-9001" are what embeddings fumble. Contextual retrieval: prepend a short model-written note to each chunk explaining what document it's from — Anthropic measured 49% fewer retrieval failures, 67% with reranking (a second pass that re-orders candidates by relevance) added.¹ Only after the ladder is exhausted do you talk about switching models.

One more scoping angle: with million-token context windows, sometimes the honest answer is no RAG at all — if the whole knowledge base fits in 200 pages, just include it (prompt caching makes the repeat cost bearable). And the reverse confusion to catch early: fine-tuning does not reliably add facts — that's Lesson 08. RAG is for knowledge; tuning is for behavior.

🧪 Lab: build RAG from scratch — and break it

The lab file is labs/0006-rag-pipeline.py — no vector database, no framework, just the pattern in plain Python. Part 1 asks about a fictional company's policies without grounding and lets you watch the confident fabrication happen.

Part 2 builds the pipeline: a 12-chunk document base, a scoring retriever, top-3 chunks fenced into the prompt with citation instructions — and the same questions now come back correct, with passage references. Part 3 breaks it: a question worded so keyword retrieval misses the right chunk, so you see a retrieval failure produce a wrong answer while the generation step works perfectly. Diagnosing that gap is the day-one skill of RAG work.

You are a senior AI engineer reviewing my work: the output of my from-scratch RAG lab (labs/0006-rag-pipeline.py) plus my written diagnosis of the Part 3 failure and my proposed fix.

Grade each item as Strong / Adequate / Missing, with one sentence of evidence:
- My Part 1 vs Part 2 comparison shows the same question answered fabricated-then-grounded, and I can explain the mechanism (the fact entered the context window).
- My Part 3 diagnosis correctly locates the failure in RETRIEVAL, not generation, with the evidence from the printed retrieved chunks.
- My proposed fix climbs the repair ladder sensibly (hybrid/semantic search, contextual retrieval, reranking) rather than jumping to "use a bigger model."
- I can articulate when I would recommend NO RAG (small corpus + long context) and why fine-tuning is not the tool for adding facts.

Be skeptical — challenge my weakest diagnosis first. Then ask me 2–3 follow-up questions a customer's data-security officer would ask. Finish with the single highest-leverage improvement.

My work follows below.

Check yourself — diagnose the pipeline

Three RAG incidents. Don't scroll up — locate the failing component from the mechanism. Wrong picks stay live.