RAG metrics: faithfulness, precision and recall

Why it matters

A retrieval-augmented system can be wrong in two places. The retriever can fetch the wrong chunks, or the generator can say things the chunks do not support. The user sees one bad answer and cannot tell which happened; neither can a single "accuracy" score. Interviewers ask for the metric vocabulary early because the names show whether you can localise a failure to a stage.

The idea

A RAG call is two stages, so there are two families of metrics: retrieval metrics score the chunks, generation metrics score the answer. Each is a number between 0 and 1, averaged over a set of questions. The names follow the Ragas convention.

Retrieval metrics

Context precision asks whether the relevant chunks are ranked high. It rewards relevant chunks near the top of the retrieved list, so a list of ten with the useful chunk at rank nine scores low even though the chunk is present. It is the ranking-quality metric.

Context recall asks whether the retrieved context contains what the reference answer needs. Each sentence of the reference answer is checked for support in the chunks, and the fraction supported is the score. A low value means the evidence was never fetched, whatever the generator did afterwards.

The classic information-retrieval metrics sit underneath and need labelled relevant chunks rather than a reference answer. Recall@k is the fraction of relevant chunks in the top k. Precision@k is the fraction of the top k that are relevant. MRR, mean reciprocal rank, is one over the rank of the first relevant chunk, averaged over questions; it rewards getting one good chunk to the top. nDCG discounts each relevant chunk by its rank and normalises against the best possible ordering, for when several chunks and their order matter.

Generation metrics

Faithfulness asks whether the claims in the answer are supported by the retrieved context. The answer is split into atomic claims, each is checked against the chunks, and the fraction supported is the score. It is the hallucination metric, and it says nothing about whether the context was right.

Response relevancy asks whether the answer addresses the question. One common method generates a few questions the answer would be a good reply to and compares them with the original; an answer that drifts, pads or refuses scores low.

Noise sensitivity asks whether irrelevant context derails the model. Add plausible but wrong distractor chunks and measure how often the answer now carries unsupported claims.

Which metric catches which failure

The failure points of RAG lesson lists where a pipeline breaks; each break moves a metric. Content missing from the corpus shows as context recall that stays low however the retriever is tuned. A retrieval miss, where the chunk exists but is not in the top k, shows in recall@k and context recall. A chunk fetched but ranked below the cut shows in context precision, MRR and nDCG. The generator ignoring good context shows as high context recall with low answer correctness. Invented claims show as low faithfulness, distractors in noise sensitivity, and a wrong format or off-topic answer in response relevancy or, more cheaply, in a code check.

	Measures	Needs a reference	Catches
Context precision	Relevant chunks ranked near the top	No, when judged against the question	Poor ranking; budget spent on noise
Context recall	Reference answer sentences supported by the chunks	Yes, a reference answer	Retrieval miss; missing content
Recall@k, MRR, nDCG	Labelled relevant chunks found and their rank	Yes, labelled chunks	Retrieval miss; chunk below the cut
Faithfulness	Answer claims supported by the chunks	No	Hallucination; prompt not bound to sources
Response relevancy	Answer addresses the question	No	Drift, padding, wrong format, refusal
Noise sensitivity	Answer survives distractor chunks	Yes, to spot the wrong claims	Model derailed by irrelevant context

Reference-based versus reference-free

Context recall, recall@k, precision@k, MRR and nDCG need a reference: a golden answer or labelled relevant chunks. Faithfulness, response relevancy and, in its judged form, context precision are reference-free: they score the question, context and answer on their own, usually with a model as judge. Reference-free metrics run on live traffic at any scale; reference-based metrics are more trustworthy and gate releases.

Reading a metric honestly

A faithfulness of 0.9 next to a context recall of 0.4 is not a good system. It is a model faithfully summarising evidence that is mostly the wrong evidence: confident answers built on missing facts. Read the pair, not the number. Averages hide segments, so split by question type and corpus area before concluding anything. A judged metric is only as reliable as its judge, which the LLM-as-judge for retrieval lesson covers.

Trade-offs

Reference-based metrics cost labelling hours; judged metrics cost model calls and inherit the judge's biases. Precision and recall pull against each other through k: a larger k lifts recall, lowers precision and spends more context budget. None of them measures whether the user was helped, which is why the rating and the trace still matter.

In an interview

Name the two stages and pin a metric to each: "I measure retrieval with context recall and precision against a golden set, and generation with faithfulness and response relevancy, because a bad answer has to be localised to a stage before it can be fixed." At junior depth, know the six names and which stage each scores. At mid depth, explain how faithfulness and context recall are computed from claims, what recall@k, MRR and nDCG reward, and the reference split. At senior depth, read pairs of metrics as diagnoses and tie reference-free scoring to production sampling. Expect "Which metric first?" (context recall: if the evidence is not there, nothing downstream matters) and "Can faithfulness be gamed?" (yes, by refusing or by saying very little).