Diagnosing a badly rated RAG system

Why it matters

"Our RAG assistant is rated 2.1 out of 5. What do you do?" is the interview question for this whole stage. The weak answer lists things that could be wrong. The strong answer is a procedure: start from evidence, narrow to a stage, measure that stage with the right metric, fix one thing, prove the fix. Interviewers use the question because it shows whether you treat a RAG system as a pipeline with observable stages or as a box that is sometimes bad.

The idea

Low ratings are a symptom. The procedure turns them into a ranked list of causes.

Start from the ratings and the traces

Pull the rated conversations with their traces: the query, the retrieved chunks with scores, the assembled prompt, the answer. If traces are not stored, storing them is the first fix. Then segment the ratings by question type, corpus area, user cohort and time. A 2.1 average is often a 4.0 on lookups and a 1.3 on comparisons. The segment that is both worst and largest is where you look first.

Sample twenty and classify

Take twenty of the worst-rated conversations from that segment and read each trace end to end. Classify each by failure point, in pipeline order:

• Missing content: the answer is not in the corpus at all.
• Retrieval miss: the right chunk exists but was not in the top k.
• Cut by budget: retrieved, but ranked below the cut or dropped by the context limit.
• Ignored by the model: in the prompt, but the answer did not use it or contradicted it.
• Wrong format: right content, wrong shape; a wall of text where a table was wanted.
• Stale: the answer was right last quarter.
• Permission: the user could not see the document needed, or saw one they should not have.

Twenty is enough: the largest bucket is obvious at that size, and reading them takes an afternoon.

Measure the suspected stage

The bucket points at a stage; the metric confirms it. For retrieval misses, run recall@k and context recall on the golden set slice for that segment. For cut-by-budget, look at context precision and the rank of the reference chunk. For ignored-by-the-model, compare context recall (high) with faithfulness and answer correctness (low). For missing content, count how many questions the corpus cannot answer. The number is your before figure.

Fix the largest bucket first

One change at a time. Fix the largest bucket, re-run the golden set, and confirm the segment moved without the others regressing. Ship behind a flag to a fraction of traffic, watch the rating for that cohort for a week, then widen. Then take the next bucket. A system tuned by three simultaneous changes cannot be explained or rolled back one piece at a time.

The common real causes

In practice the buckets fill in a predictable order. Parsing loss comes first: tables flattened to word soup, PDF columns interleaved, so the chunk that should answer is unreadable. Chunking is next: chunks that split a procedure in half, or so large that the embedding averages three topics into none. No reranker: the right chunk comes back at rank seven and the budget cuts at five. The prompt does not bind to the sources: the model is given chunks but not told to answer only from them and cite, so it blends training memory with context. No handling of out-of-scope questions: the system answers everything, and the confident wrong answers to questions it should have declined are the ones users rate lowest. Four of the five sit before the model, which is why swapping the model rarely moves a 2.1.

	Parsing loss	Chunking	No reranker
Symptom	Answers wrong on anything from a table or multi-column PDF	Half a procedure; answers that stop mid-step	Right chunk appears in traces at rank six to ten
Test	Open the stored chunk text for ten bad cases and read it	Context recall by document type; chunk length histogram	Recall@20 high while recall@5 low; MRR below 0.5
Fix	Layout-aware parser; tables to markdown; re-index	Structure-aware splitting with overlap; re-index	Cross-encoder rerank of top 30 to top 5
Effort	Days, plus a full re-index	Days, plus a full re-index	One to two days, adds 50 to 200 ms per query

Narrating it in 45 minutes

Spend the first five minutes asking what is logged and what the rating is attached to. Spend ten on segmenting and sampling, saying out loud how you would classify the twenty. Spend fifteen on the top two buckets: the metric that confirms each, the fix, and the before-and-after number you expect. Spend ten on shipping: the golden set as the gate, the flag, the cohort rating. Keep five for follow-ups, and name the order of the real causes unprompted.