Building an eval set from production traces

Why it matters

The first eval set is written from the spec before launch, and it measures what the team imagined. Production disagrees within a week: users paste error logs, ask two questions in one, type in another language, and hit the case nobody wrote. Traces hold all of that, with the context the system saw and the output it gave: the richest source of cases there is. The Building a golden set lesson covers the retrieval version with reference chunks; this one is the general method for any model-backed feature. Interviewers ask it as "how does your eval set keep up with the product?"

The idea

A trace is the record of one interaction: input, retrieved context, tool calls, output, latency, cost, and whatever signal the user left. Turning traces into an eval set is a pipeline with five stages: sample, convert, label, deduplicate, version.

Sampling

Three strategies, used together. Failure-signal sampling pulls traces where something went wrong: a thumbs down, a retry of the same question, an escalation to a human, a session that ran long, a judge flag on the production sample, a guardrail block. Segment sampling takes a fixed number from each slice you care about (product line, language, customer tier, question type) so a thin segment is not drowned by the busy one. Random sampling keeps the set honest: a set made only of failures measures the hard tail and says nothing about the typical call, and a prompt tuned only on failures regresses the easy majority.

From trace to case

A case needs an input, the context the system should see (or a way to reproduce it), and a statement of what good looks like, which is not always a reference answer. For a refund question it may be a rubric: cites the policy, promises no money, under 120 words. For a coding task it is a test. For an agent it is a goal condition and a budget. Writing a reference answer for every case is the expensive mistake; writing the check that would have caught the failure is the cheap one. Keep the trace id on the case so you can go back to it.

case 0412   from trace 7f3a   segment=billing/en   type=wrong-policy
input:    "charged twice this month, can I get the second one back?"
context:  account summary (synthetic, PII stripped)
expect:   rubric policy-v3; must_not promise a refund; tool=open_ticket

Labelling

Someone must say what the right behaviour was. For failure traces the user often already said it; for random traces an expert reads and labels. Two labellers on a sample give an agreement rate; below about 80 percent the labelling guide is the problem, not the labellers. Tag each case by segment and failure type so a score drop can be sliced later.

Deduplication and privacy

Traces repeat. Ten users asking the same question are one case, or the set measures that question ten times. Cluster by embedding or normalised text and keep one per cluster, with the count as a weight. Before anything enters the set, strip personal data: names, emails, account numbers, pasted documents. The set lives in the repository and is read by everyone; a trace is not.

Versioning, held-out and refresh

The set carries a version number and a changelog, and every score is reported against a named version. A third of the set is held out: nobody tunes a prompt against it, it runs only at release, and the gap between the tuning score and the held-out score is the overfitting measure. Refresh on a cadence and on drift: when the segment mix shifts, a feature ships or the model changes, add fresh traces and retire cases the product no longer covers.

Size

Hundreds, evaluated on every change, beat thousands evaluated never. A few hundred cases is enough to see a five-point move, and a set that runs in minutes is one nobody skips. Keep a fast slice for every pull request and the full set for releases.