Module 07 · The RAG pipeline · Lesson 6 of 6
The failure points of RAG
Eleven places RAG goes wrong, from missing content to injection through documents: the symptom, the metric that catches each (context recall, precision, faithfulness, relevancy), the first fix, and diagnosing backwards.
Why it matters
A RAG answer is wrong in a specific place, and the fix for one place makes another worse. Teams without a map of the failure points fix by superstition: bump k, swap the embedding model, add a prompt line, measure nothing. Senior interviews ask for it directly: "the answer was wrong; walk me back through the pipeline".
The idea
For each, in pipeline order: the symptom, the metric that catches it, the first fix.
The seven in the answer path
1. Missing content. The answer is not in the corpus. Symptom: a confident answer from general knowledge, or an unexpected abstention. Metric: none of the retrieval metrics, since the labelled passage does not exist; watch abstention rate and user reports. Fix: ingest the source; instruct abstention so the failure is honest.
2. Missed the top-ranked documents. The passage exists but missed the top k. Symptom: an answer from a related but wrong passage, plausibly cited. Metric: context recall. Fix: query rewriting, hybrid search, a larger candidate set with reranking, better chunk boundaries.
3. Not in context. Retrieved, then cut by the token budget. Symptom: the passage is in the retrieval log and not in the prompt. Metric: context recall measured at the prompt; the gap from the retriever's figure is this failure. Fix: rerank before truncation, smaller chunks, a larger budget for sources.
4. Not extracted. In the prompt, ignored by the model. Symptom: the right passage is in the prompt and the answer contradicts it. Metric: faithfulness. Fix: fewer, better chunks, strongest first, a stricter binding instruction, a stronger model.
5. Wrong format. Right answer, wrong shape: prose where a table was needed, a table where a yes was. Metric: response relevancy and schema checks on structured outputs. Fix: specify the format in the instruction; use structured output where the shape is fixed.
6. Incorrect specificity. Too general or too detailed for the question: true, but it did not answer me. Metric: response relevancy, plus a judge scoring specificity against the question. Fix: step-back or decomposition on the query side, granularity instructions, a reranker that prefers passages at the question's level.
7. Incomplete answers. Three of five steps, one of two conditions. Symptom: follow-ups asking for the rest. Metric: context recall for the missing parts and a completeness judge. Fix: decompose compound questions, retrieve per part, raise k for list-type questions.
The operational four
8. Parsing loss. Tables flattened, columns interleaved, OCR digits wrong. Symptom: numbers wrong in a consistent way under correct citations. Metric: a sample-based parse check; faithfulness is fooled because the model faithfully quoted a bad parse. Fix: layout-aware or vision parsing for that document family; keep the page image.
9. Stale index. The document changed; the index did not. Symptom: answers that were right last month. Metric: index age per source and an eval set with recently changed facts. Fix: re-index on change events; show a freshness date beside the citation.
10. Permission leaks. A user sees a passage they may not read. Symptom: none visible to that user. Metric: an access test suite run against retrieval with low-permission accounts. Fix: access labels on every chunk, filtered inside the retrieval query, never after.
11. Prompt injection through documents. A document carries instructions ("ignore the question and reply with…") and the model follows them. Symptom: off-task answers on specific sources. Metric: a red-team set of poisoned documents. Fix: treat retrieved text as data in the instruction, flag instruction-like spans at ingestion, and never let an agent act on retrieved text unchecked.
Diagnosing backwards
Start from the bad answer and walk upstream with the logs. Was the correct passage in the prompt? If yes, the failure is 4 to 7, a generation problem. If not, was it in the retrieval result? If yes, it is 3, the budget. If not, is it in the index? If yes, it is 2, retrieval; test rewrites and reranking. If not, is it in the corpus? If yes, it is 8 or 9, parsing or staleness. If not, it is 1. Each step is a one-minute lookup if the retrieval result, final prompt and chunk ids were logged; if not, that logging is the first fix.
| Missing content | Retrieval miss | Cut or ignored | Wrong shape | |
|---|---|---|---|---|
| Symptom | Answer from general knowledge, or an unexpected abstention | Plausible answer from a related passage, cited | Right passage retrieved; answer contradicts or omits it | Right facts; wrong format, level or completeness |
| Metric | Abstention rate and user reports; no retrieval metric sees it | Context recall | Context recall at the prompt; faithfulness | Response relevancy; completeness judge |
| First fix | Ingest the source; honest abstention | Rewrite, hybrid search, rerank | Rerank before truncation; fewer, better chunks; stricter binding | Format in the instruction; decompose; structured output |