Module 09 · Measuring retrieval · Lesson 2 of 4
Building a golden set
How to assemble 50 to 300 question, answer and chunk triples from logs, tickets and reviewed synthetic questions, label them with experts, version them next to the code, refresh them, and avoid tuning on your own test.
Why it matters
Most RAG teams tune by feel. Someone changes the chunk size, asks the three questions they remember, and ships. A week later someone else changes the prompt and the first fix quietly regresses, and nobody knows. A golden set ends that: a fixed list of questions with known good answers and the chunks that support them, run on every change, so a regression is a red line in CI rather than a complaint from a customer. You cannot improve what you cannot measure, and the golden set is where measurement starts.
The idea
A golden set is a collection of triples: a question, a reference answer, and the reference chunks the answer should be drawn from. Fifty triples is enough to start; three hundred covers a mid-sized corpus well. The reference chunks make retrieval metrics possible (recall@k, context recall); the reference answer makes generation metrics possible (answer correctness, and faithfulness checked against the right evidence).
Where the questions come from
Logged user questions are the most realistic source. Pull them from traces, de-duplicate, strip personal data, and keep the phrasing exactly as typed, typos included. Support tickets add the questions people took to a human after the system failed them, a direct sample of the hardest cases. Synthetic questions come from giving a model a chunk and asking for questions it answers; they are cheap and reach corners no user has visited, but they are phrased like the document, which flatters the retriever, so a human must read each one, cut the trivial and reword the ones that quote the source. Adversarial and out-of-scope questions round the set out: ambiguous questions, questions the corpus cannot answer, questions that span two documents, injection attempts. The right reference answer for an out-of-scope question is a refusal.
Coverage
Tag every question by type (factual lookup, comparison, procedure, out of scope) and by corpus area (product line, region, document family). Then look at the counts. A set of 200 lookups and three comparisons measures lookups. Fill the thin cells deliberately, with synthetic questions if the logs lack them; the tags are also how you segment a drop later.
Labelling
Experts write or verify the reference answers and mark the reference chunks. That is the expensive step, roughly five to fifteen minutes per triple, so budget forty to sixty hours for 300 triples and protect that time. Two labellers on a sample of 30 give an agreement rate; below about 80 percent the labelling guide is unclear and any metric built on it will be noise.
| Logged questions | Synthetic from documents | Expert-written | |
|---|---|---|---|
| Realism | Highest: real phrasing, typos, context missing | Low: phrased like the source, flatters retrieval | Medium: well-formed, often harder than real |
| Cost | Low to collect, needs de-duplication and PII stripping | Very low to generate, moderate to review | High: expert hours per question |
| Coverage | Follows yesterday's traffic; thin on new corpus areas | Can target every chunk and corner | Targets what experts think matters |
| Bias | Over-represents frequent users and old features | Over-represents document vocabulary | Over-represents interesting edge cases |
| Reach for it | The backbone once there is traffic | Before launch and to fill thin cells | Adversarial, compliance and out-of-scope items |
Versioning and refreshing
The set lives in the repository next to the code, as JSON or YAML, with a version number and a changelog; a score only means something against a named version. When the corpus changes, reference chunks go stale: documents are deleted, split or rewritten, and an item whose chunk no longer exists must be re-labelled or retired. Refresh with every major corpus update and lightly each month, adding recent logged questions so the set tracks what users ask now.
The leakage trap
If you tune chunk sizes, prompts and rerankers against the whole golden set, it stops measuring generalisation and starts measuring how well you memorised it. Hold out a slice, a third is typical, that nobody tunes against and that only runs at release time. When the held-out score and the tuning score diverge, you have overfitted the pipeline to your own test.
Small and often
A set of 60 questions that runs on every pull request beats a set of 2,000 that runs once a quarter. The value is in the diff: this change moved context recall from 0.71 to 0.64 on procedure questions. Keep the fast set cheap enough that nobody skips it, and keep the larger set for releases.