Chain of thought and thinking effort

Why it matters

A model answering in one pass has one shot at every token: it cannot go back, check a sub-result or change its mind. For arithmetic, planning and anything with several dependent steps, that produces confident wrong answers. Chain of thought was the discovery that letting the model write its working first moved the hard part out of a single token prediction and into a sequence it could build on. Reasoning models made that behaviour native, and the question for an engineer is no longer "should I ask for reasoning" but "how much thinking should this call buy". Interviewers ask because thinking tokens are the largest discretionary cost in a modern LLM system.

The idea

Reasoning in the output

The original trick was a prompt: "think step by step before you answer". It worked because every generated token becomes context for the next one. A model that writes "17 times 23: 17 times 20 is 340, 17 times 3 is 51, total 391" has turned one hard prediction into three easy ones, each conditioned on the last. Intermediate steps are scratch memory, and the model's own text is the only scratch memory it has.

Thinking as a phase

Current models internalise this. Before the visible answer they produce a thinking span: tokens generated the same way, in the same decode loop, but marked so the application can hide or discard them. Training rewards thinking that leads to correct final answers, so the span is not decorative; it is where the search happens, one predicted token at a time.

The effort dial

The spend is now a parameter. On the current Claude lineup adaptive thinking is always on and an effort setting steers how much of it the model does; on OpenAI's line a reasoning-effort setting runs from none to max. The model decides where to spend within the budget; the setting moves the ceiling. High effort on a hard planning problem can produce tens of thousands of thinking tokens before the first answer token.

What it costs

Thinking tokens are output tokens. They are billed at the output price, several times the input price, and they are generated serially, so latency grows with them directly. A call that answers in 200 tokens at low effort might spend 6,000 thinking tokens at high effort: thirty times the output cost, and a wait for the first visible token measured in tens of seconds rather than under one.

When to spend

Spend on maths and anything numeric, on planning a multi-step task, on multi-step tool use where the model must decide what to call next and interpret the result, and on ambiguous instructions where it must weigh interpretations. Do not spend on classification, extraction, reformatting, short conversational turns or anything where the answer is a lookup in the prompt: there is nothing to reason about and the tokens are waste. The practical rule is to default low, measure on an eval set, and raise effort only where accuracy moves.

The expensive extreme

Self-consistency samples several independent reasoning paths and takes the majority answer. It lifts accuracy on problems with one checkable answer and multiplies cost by the sample count, five to ten times, so it belongs in offline grading and single high-stakes decisions, not in request paths. A reasoning model at high effort captures much of the same gain inside one call.

	No thinking	Low effort	High effort	Voting (self-consistency)
Latency	First token in well under a second	Adds a few hundred tokens of delay	Seconds to minutes before the answer starts	N full calls; parallel if you pay for it
Cost	Answer tokens only	Small multiple of the answer	Ten to thirty times the answer in output tokens	N times a high-effort call
When it pays	Lookup, classification, extraction, chat	Light judgement, short tool use	Maths, planning, long tool chains, ambiguity	Offline grading, one-off high-stakes decisions
Failure mode	Confident errors on multi-step problems	Stops thinking too early on hard inputs	Overthinks trivial inputs; budget blown	Majority of wrong paths agree on a wrong answer

Trade-offs

Visible reasoning is not a faithful trace. The thinking span is text the model produced on its way to an answer, trained to be useful rather than true; the real computation is in the weights, and a model can write a plausible chain and answer for another reason. Do not build safety or audit logic on it, and do not show it to users as an explanation. In multi-turn agents earlier turns' thinking is usually dropped, so a plan made while thinking does not persist unless written into the answer or a tool call. Effort is set per call, so a request that mixes a trivial lookup with a hard sub-problem pays high effort for both; split it into two calls with two settings.

In an interview

Say: "Thinking tokens are output tokens; I default to low effort, raise it where an eval shows accuracy moving, and keep voting for offline grading." At junior depth, explain why writing steps helps and that reasoning is billed as output. At mid depth, describe the effort parameter, the latency and cost arithmetic, and the spend and no-spend lists. At senior depth, discuss unfaithful reasoning, effort per route in a system, and when self-consistency is worth ten times the cost. Expect: "Would you show the chain of thought to the user as an explanation?" The answer is no, with the faithfulness reason.