Reasoning models and test-time compute

Why it matters

For a decade the way to get a better answer was a bigger model. Reasoning models add a second lever: let the same model think longer at inference time. That changes the cost model of an LLM system. A request is now priced partly by thinking tokens you did not write and may never read, and latency depends on how hard the model decided the problem was rather than on answer length. Interviewers in 2026 ask about effort settings for the reason they once asked about caching: an engineer who cannot say when thinking pays for itself ships a product that is slow and expensive on easy requests and wrong on hard ones.

The idea

A reasoning model is a model post-trained, as the previous lesson described, to generate a stretch of thinking tokens before its answer: working, checking, backtracking, in ordinary text that the API may return in full, summarised, or not at all. Mechanically nothing is new; they are output tokens from the same forward passes. What is new is that the model learned to use them, and that accuracy on hard problems rises with their number.

Test-time compute as a scaling axis

Pre-training scaling says loss falls with training compute. Test-time compute scaling says accuracy on a given task rises with inference compute: more thinking tokens, or several sampled attempts with a verifier choosing among them. The curve is steep at first, then flat. A problem that needs ten steps of working is unlocked by a few thousand tokens; a hundred thousand more barely move it. Cost, meanwhile, is linear in tokens all the way up. The gap between those two curves is where routing decisions live.

Adaptive thinking and effort

Current models decide per request how much to think and expose an effort parameter that biases that decision. Anthropic's current Claude models have adaptive thinking always on, steered by an effort setting; OpenAI's current line exposes a reasoning-effort setting that runs from none to max. Low effort on a trivial prompt yields almost no thinking tokens; high effort on a hard one may yield tens of thousands. Provider-neutral: thinking depth is a request-level dial, and choosing its position is part of system design.

When thinking helps and when it is wasted

Thinking pays on tasks with many dependent steps and a checkable end: maths and quantitative reasoning, code generation and debugging, multi-step planning, and tool-heavy agent loops where a wrong early call compounds. It is wasted on tasks a single pass already handles: classification, entity extraction, summarising a short text, routine chat, rewriting. On those it adds latency and cost and can lower quality, because a model told to think hard about a trivial label sometimes talks itself out of the obvious one.

The arithmetic

Thinking tokens are billed as output tokens, the expensive kind, typically several times the input price. Take a request with 2,000 input tokens and a 300-token answer at a representative $3 per million input and $15 per million output. With no thinking it costs about 1 cent. At high effort producing 8,000 thinking tokens it costs about 13 cents, twelve times more, and the time to the first visible token moves from under a second to tens of seconds, because nothing is shown until the thinking ends. At a million requests a day that difference is the infrastructure budget. The counter-move: a smaller model at high effort often beats a larger one at none on hard tasks, for less money, so the real choice has two axes, model size and effort.

	No thinking	Low effort	High effort
Time to first visible token	Under a second	A few seconds	Tens of seconds
Cost per request	Prompt plus answer	Plus hundreds to a few thousand tokens	Plus thousands to tens of thousands
Accuracy on hard tasks	Baseline	Most of the gain	The last few points
Fit	Classification, extraction, chat	Code edits, structured multi-step answers	Maths, hard debugging, long agent plans

Trade-offs

More thinking is not monotonic in quality: past the knee it stops helping, and on easy tasks it can hurt. Thinking tokens are often hidden or summarised, so your logs and context budgets must account for tokens you cannot see. Latency becomes variable per request, which breaks fixed timeouts; stream, cap the thinking budget, and set the timeout above the cap's worst case. Thinking also inflates the output limit you must request, and in an agent loop the provider usually drops earlier turns' thinking, so do not assume the model remembers what it worked out two turns ago.

In an interview

The line to have ready: "Match effort to the task and route. Classify the request first, send easy traffic to a fast model with thinking off, and reserve high effort for the hard, checkable work where accuracy pays for the tokens." At junior depth, know what a reasoning model does and that thinking is billed as output. At mid depth, explain the accuracy-against-tokens curve, effort settings, and the cost and latency arithmetic. At senior depth, design the router, set each workload's effort from accuracy against cost on a labelled sample of your own traffic, and cover budget caps, timeouts and hidden tokens. Expect: "How do you pick the effort for a workload?" (measure accuracy at each level and take the cheapest that meets the target). "What happens to thinking tokens across turns?" (they are dropped; only the visible answer and tool calls persist).