LLM-as-judge for retrieval

Why it matters

Faithfulness and relevancy are judgements, not string comparisons. A human can read an answer against its chunks and say whether every claim is supported, but a human costs minutes per item, a golden set has hundreds, and a production day has thousands. The practical answer is to make a model do the reading: give it the question, the context, the answer and a rubric, and ask for a score. That is LLM-as-judge. Interviewers ask about it because it is easy to deploy badly: an uncalibrated judge produces confident numbers that mean nothing, and the team optimises against them.

The idea

A judge is an ordinary model call whose prompt carries a rubric, the items to score and an output format. For retrieval work there are three judgements worth automating: is each claim in the answer supported by the context (faithfulness), does the answer address the question (relevancy), and is each retrieved chunk relevant to the question (chunk relevance, the building block of context precision).

The rubric is the prompt

The rubric must say what each level means, what to ignore (style, length) and what to do when the context is empty. Ask for the reasoning before the verdict so the verdict is conditioned on it, and ask for structured output so code can read it.

You grade one answer from a documentation assistant.
The question, the retrieved chunks and the answer follow.
Step 1: list every factual claim in the answer.
Step 2: mark each claim SUPPORTED, CONTRADICTED or NOT IN
        CONTEXT, naming the chunk id that applies.
Step 3: faithfulness = supported claims / all claims.
Ignore tone, length and formatting.
If the answer declines to answer, return N/A.
Reply as JSON with claims, verdicts, faithfulness, note.

Pairwise versus absolute

An absolute judge scores one answer on a scale. A pairwise judge sees two answers to the same question and picks the better one. Pairwise is more reliable for comparing two versions of a pipeline, but it yields a win rate rather than a number to track over time. Use absolute scoring for faithfulness and chunk relevance, where the rubric is close to a checklist, and pairwise when choosing between two prompts or rerankers.

Known biases

Judges inherit the habits of the models behind them. Position bias: in a pairwise prompt the first answer wins more often, so swap the order and judge twice. Verbosity bias: longer answers score higher, so the rubric must say length is not a factor. Self-preference: a model rates its own outputs higher, so judge with a different model from the one that generated. Leniency: judges round up, calling a partly supported claim supported, so the rubric needs a verdict per claim rather than one overall feeling.

Calibration

A judge is itself a model under test. Take a few hundred items with human labels, run the judge, and report agreement: percent agreement for binary verdicts, a correlation for scales. Above roughly 85 percent the judge is useful; around 60 percent it is adding noise and the rubric, the model or the labels need work. Report it next to every judged metric and re-calibrate whenever the judge model or the rubric changes. Read the disagreements by hand; most are rubric ambiguity rather than model failure.

Which model judges, and what it costs

Use a stronger model than the one generating, or at least a different one. Each evaluation costs the chunks plus the answer plus the rubric, typically one to three thousand input tokens and a few hundred output. At batch prices that is small for a golden set of 300 and affordable in production, which leads to sampling.

Judges in production

On live traffic you cannot judge every call, and you do not need to. Sample one to five percent, judge asynchronously, and plot the faithfulness rate by day and by segment; a drop is an alarm before users complain. A judge can also run inline as a guardrail, blocking an answer with unsupported claims, but that adds a model call of latency to every response, so reserve it for high-stakes surfaces.

When code beats a judge

Some checks are deterministic. Did the answer cite a chunk id? Does the output parse against the schema? Does it contain the required product code or date? Did the system refuse when the question was tagged out of scope? Those are code: they cost nothing and never disagree with themselves. Reach for a judge only when the question needs reading.