Benchmarks and why they mislead

Why it matters

Every model launch comes with a table: knowledge, reasoning, maths, coding, agent task suites, each a percentage next to a competitor's. Leaderboards rank the same models on the same tables. Engineers read them as a buying guide, and the interviewer wants to know whether you do too. The honest position: a benchmark says something about general capability and almost nothing about your task, and the only number with decision weight comes from your own eval set.

The idea

What a benchmark is

A public benchmark is a fixed set of items with a scoring rule: multiple-choice questions on general knowledge, maths problems with a checkable answer, coding tasks with hidden tests, agent task suites in a sandbox with a goal condition. They are useful because they are shared: every model runs on the same items, and a leaderboard orders the results. A second kind of leaderboard collects human preference votes on anonymous pairs of answers, which measures what people like rather than what is correct.

Contamination

The benchmarks are on the internet, and so is the training data. Test items, their paraphrases and their solutions leak into the corpus, and a model then scores on memory rather than skill. Vendors filter, but no filter catches a rewritten item, and the newest benchmarks are the least contaminated only until they are published. A high score on a years-old benchmark is weak evidence; the same model may fail a fresh item of the same shape.

Saturation

When the top models all score near the ceiling of a benchmark, it stops ranking them; the differences are inside the noise of the items. Harder benchmarks are written and the cycle repeats. A table where every model sits within a few points at the top tells you nothing.

The gap to your task

A benchmark measures a general skill under its own conditions: its prompt format, its scoring rule, its item mix. Your task is a support agent in Portuguese over a product the model has never seen, with tools and a latency budget. Nothing in a general reasoning score predicts how a model handles your tool schema or your refusal policy. Rankings routinely flip between a public benchmark and a task-specific set: the model that leads on reasoning comes third on your extraction task because it over-explains, and the cheaper model wins because it follows the format.

Selection and variance

The vendor chooses which benchmarks to show, which prompt format to use, how many samples to take, and whether to report the best of several runs. A model card's table is an argument, not a measurement. Scores also move by a few points across runs and prompt wordings, so a small lead between two models is noise unless the variance is reported, and it rarely is.

How to use them

Benchmarks are for shortlisting, not deciding. Use them to rule out a model that is clearly below the bar on a capability you need, to notice that a small open-weight model is close enough to be worth testing, and to compare price against general level. Then run the shortlist on your own eval set, with your prompt, your tools and your latency budget, and decide on that. The decision weight runs: your eval set first, then an A/B test, then an independent benchmark on a fresh suite, then the vendor's table.

Reading a model card table

Ask of every row: is the comparison model the current version or an old one; are the prompt format and sample count the same for every column; is a variance given; when was the benchmark published relative to the training cutoff; and does the task resemble mine at all. Count the rows where the vendor's model wins and compare it with the number of benchmarks that exist; a table with only wins is a selection.

	Public benchmark	Vendor-reported	Your eval set
Relevance to your task	Low: general skill, its own format	Low, and chosen to flatter	Exact: your prompt, tools and users
Contamination risk	High once published	High, plus selection of what to show	None while the set stays out of training data
Cost to you	None to read; moderate to re-run	None	Hours to build, minutes per run
Decision weight	Shortlist only	A claim to verify	The decision