Eval types: unit, LLM-as-judge, human, A/B

Why it matters

"How would you evaluate that?" has four honest answers, and most candidates give one. A string check cannot tell whether an answer is kind; a judge cannot tell whether users bought more; a human cannot read ten thousand answers; an A/B test cannot tell you why. Each layer answers a different question at a different price, and a working eval stack uses all four in the right proportions. Interviewers listen for whether you know which question you are asking before you pick the tool.

The idea

Layer one: deterministic checks

Code that reads the output and returns pass or fail. Exact match against a reference for classification and extraction. Schema validation for structured output. A regex for a required product code or a forbidden phrase. "Contains a citation" for a grounded answer. For a coding agent, the tests pass or they do not. These cost nothing, run on every call and never disagree with themselves. Their limit is reach: they check form, not meaning, and a wrong answer in a valid schema passes. Reduce every question you can to a check; a judge asked whether output is valid JSON wastes a model call.

Layer two: LLM-as-judge

A model reads the output against a rubric and scores it. Pointwise scoring grades one answer on a scale or a checklist. Pairwise scoring shows two answers and picks the better, which is more reliable for comparing versions and useless for tracking one number over time. The judge must be calibrated: a few hundred items with human labels, an agreement rate reported next to every score, re-run whenever the rubric or the judge model changes. The LLM-as-judge for retrieval lesson covers the rubric and the four biases in depth; the general point is that a judge is a second model under test, and its number means nothing until agreement is known. Cost is a model call per item, cheap for a set of hundreds and affordable for a sample of production.

Layer three: human review

Experts label items. This is the reference: judges are calibrated on human labels, and the expected behaviours in the set come from humans. It is slow, minutes per item, so it runs on hundreds rather than thousands, and it needs an annotation queue, a labelling guide and an agreement check, because two experts disagree more than teams expect. Spend human time where it compounds: labelling the set, reading the judge's disagreements, and reviewing a weekly sample of production.

Layer four: online experiments

Split real traffic between two versions and compare a product metric: resolution rate, retention, revenue, escalations to a human. This is the only layer that measures what users do rather than what a rubric says, and it is the final word on a change. It is also slow (weeks to significance), expensive (real users see the worse arm) and blind to causes: it tells you B lost, not why. Run it after the offline layers say B should win, as confirmation, never first.

Which layer for which question

"Is the output well formed?" is code. "Is the answer faithful to the context?" is a calibrated judge. "What does good look like for this new feature?" is humans. "Did the change help the business?" is A/B. The mistake runs both ways: a judge where code would do wastes money; code where a reader was needed passes wrong answers.

Combining them

The pattern: code on every call, in production and in CI. The judge on the whole eval set and on a one to five percent sample of production. Humans on the set when it is built, on the judge's disagreements, and on a small weekly sample. A/B on the few changes big enough to matter. Each layer feeds the next: human labels calibrate the judge, judge flags become cases, and A/B results check that the offline score still tracks the business.

	Unit check	LLM judge	Human review	A/B test
Cost per item	Near zero	A fraction of a cent to a few cents	Minutes of expert time	Real users on the worse arm for weeks
Scale	Every call	The full set; production sampled	Hundreds	Needs enough traffic for significance
Reliability	Exact for what it checks	Good once calibrated; drifts with the model	The reference, with its own disagreement rate	Statistically sound, blind to causes
What it answers	Is the form right?	Does it read as good?	What does good mean?	Did users benefit?