Evaluator-optimizer

Why it matters

A model's first draft is a sample, not its best work. Ask it to check that draft against stated criteria and it finds problems it did not avoid, because judging is an easier task than producing. Evaluator-optimizer turns that into a loop: generate, evaluate, feed the critique back, regenerate, until the evaluator passes or the budget is spent. It is how code tools get to green tests, how translation services hold a glossary, how writing products enforce a brand voice. Interviewers ask about it because the pattern is easy to describe and hard to make pay: an evaluator that is lenient or biased burns money while reporting success.

The idea

The loop

Two roles. The generator produces a candidate from the task and, on later rounds, from the previous candidate and the feedback. The evaluator scores the candidate against explicit criteria and returns a verdict and feedback: pass or fail, what failed, and what to change first. Code runs the loop, keeps the best candidate so far, and stops on pass or at the iteration cap. The criteria are the whole pattern. If you cannot write them down, the evaluator has nothing to judge against and the loop converges on whatever the critic happens to like.

What the evaluator can be

A program, where one exists. Tests, type checkers, linters, schema validators, a glossary diff, a word-count rule. Deterministic, free, never lenient. When a program can check a criterion, use it; a model judge is for what programs cannot check. A rubric prompt, for qualities such as tone, faithfulness to a source, or coverage of required points. It receives the candidate, the criteria as a numbered list and the source material, and returns a structured verdict. A mixed evaluator, the usual production shape: program checks first, cheap and hard; the model judge only on candidates that pass them.

{
  "pass": false,
  "score": 3,
  "failures": [
    "Glossary term ledger rendered two ways (lines 4 and 11)",
    "Register too informal for a contract (paragraph 2)"
  ],
  "fix_first": "Use the glossary form of ledger throughout"
}

Where it earns its cost

Three conditions. The criteria are clear enough to write down. A critic can detect the failure, often even when the generator cannot avoid it: a dropped clause, a failing test, a banned word. And the generator improves with feedback, which it does when the feedback is specific and not when it is a score alone. Translation against a glossary, code against tests, writing against a style guide, documents against a checklist: all three hold. Open-ended creative work rarely meets the first.

Loop limits and cost

Each round is a generate call plus an evaluate call. With a 2,000-token candidate, 300 tokens of feedback and a 1,500-token source, round one reads about 3,500 tokens for generation and 4,000 for evaluation, and every later round adds the previous candidate and the feedback to the generator's input. Three rounds run near 25,000 tokens against 3,500 for a single draft, seven times the cost, so cap the rounds at two or three, stop early on pass, and ship the best-scoring candidate if the cap is hit, with its verdict attached. Track the pass rate per round: if most candidates pass at round two and almost none improve at round three, the cap is one too high.

The judge's biases

A model judging a model has known distortions. Self-preference: a model rates its own output, or output in its own style, higher than equivalent output from another model. Leniency: judges pass more than humans do, especially the first candidate and long, confident text. Position and length effects: candidates shown first or written longer score higher. Mitigate by using a different model, or a differently prompted one, as judge; by asking for failures before a score; by requiring the exact span that fails; and by calibrating the judge against a few hundred human-labelled candidates before trusting its pass rate. An uncalibrated judge is a random number with good manners.