Inference anatomy: prefill, decode, KV cache

Why it matters

Every price, every latency figure and every context limit a provider quotes falls out of two phases that happen inside the accelerator when a request is served. Engineers who have not seen those phases argue about model choice when the bill is really a cache size, and promise a 1M-token window to product without knowing that a filled window has to live in memory for the whole conversation. Interviewers ask about prefill, decode and the KV cache because it is the shortest route to finding out whether a candidate understands what they are paying for, and because every serving trick in this module is a fix for one of the three.

The idea

One request, two phases

A transformer generates one token per forward pass, and each pass attends over every token before it. Serving a request therefore splits into two phases with different economics.

Prefill takes the whole prompt, system text, history, retrieved documents and the user's turn, and runs it through the model in one parallel pass. Every prompt token is processed at once as one large matrix multiplication, which keeps the accelerator busy on arithmetic: prefill is compute-bound. Its duration grows with prompt length, and nothing is emitted until it finishes, so prefill sets time to first token.

Decode then produces the answer one token per step. Each step does a small amount of arithmetic for a single new token but must read the model's weights and the whole cache of earlier tokens out of accelerator memory. The reading dominates, so decode is memory-bandwidth-bound: step time is close to constant whatever the token, and it sets tokens per second. Output latency is output tokens divided by that rate.

The KV cache

Attention needs, for every earlier token, the key and value vectors each layer computed for it. Recomputing them at every step would make each decode step as expensive as a fresh prefill of the whole sequence. So the server keeps them. The KV cache holds the keys and values for every token in the sequence, per layer, in accelerator memory, for as long as the request lives. Prefill writes the prompt's entries in one go; each decode step reads all of them and appends one more.

The cache grows linearly with sequence length and its per-token size is fixed by the architecture: two vectors per layer, each the width of the key-value heads. For a plausible 70-billion-parameter layout with grouped-query attention that is around 300 kilobytes per token. A 10,000-token conversation holds about 3 gigabytes of cache; a conversation that fills a 1M-token window needs roughly 300 gigabytes, more than the weights themselves and more than one accelerator holds. This is why long contexts cost memory before they cost compute, and why providers price and rate-limit them separately.

Why output tokens cost more

The asymmetry on the price list follows from the phases. An input token is processed once, inside a parallel pass that uses the hardware well. An output token needs its own sequential step, during which the accelerator streams gigabytes of weights and cache through memory to produce one token's worth of work. The server claws some of that back by batching many requests' decode steps together, which the next lesson covers, but a decode step still occupies far more of the machine's scarce resource per token than prefill does. Output being listed at a multiple of the input price is the hardware showing through.

The arithmetic of a request

Three numbers describe any request: tokens in, tokens out, and the cache size at the end.

request: 8,000 tokens in, 500 tokens out
prefill: 8,000 tokens in one pass            -> sets time to first token
decode:  500 steps, each reading the cache   -> 500 / tokens per second
cache:   8,500 tokens x per-token size       -> held until the last token

Prefill time is roughly proportional to tokens in. Decode time is tokens out over a rate that itself drifts down as the cache grows, because each step reads more. A request with 100,000 tokens in and 50 out is almost all prefill; a request with 500 in and 5,000 out is almost all decode. Thinking tokens are decode: a model that reasons for 4,000 tokens before answering spends that time in the slow phase and is billed for it as output.

What this means for an API user

You cannot see the accelerator, but the phases leak through the price list and the latency. Prefill cost is what prompt caching attacks: if the first 30,000 tokens of a request are identical to a recent one, the provider reuses their KV entries instead of recomputing them, so you pay a cache-read price and skip most of the prefill. That works only because the cache is a prefix structure built left to right, which is why the stable part of a prompt must come first. Decode cost is what output limits, terse formats and the effort setting attack. Memory cost is what long-context pricing and per-request token caps attack. Every lever in the rest of this module maps onto one of the three.

Trade-offs

The KV cache trades memory for time, and memory is the scarcer resource on an accelerator. Keeping a long conversation's cache alive between turns makes the next turn fast but holds gigabytes hostage; dropping it frees the memory at the price of a full prefill next time. Batching decode steps raises throughput while each request's tokens arrive a little slower. A bigger window is a marketing number until you ask what a filled one costs: fewer concurrent requests per machine, a longer prefill, and a decode that slows as the cache grows. None of this is visible from outside, so a feature that quietly doubles its prompt doubles prefill and cache on every call and nobody notices until the invoice.

In an interview

Open with the two phases: "Serving splits into prefill, which processes the prompt in one compute-bound pass and sets time to first token, and decode, which emits one token per memory-bound step and sets tokens per second; the KV cache lets decode skip recomputing attention, and it is why long contexts cost memory." At junior depth, know the two phases and that output tokens are dearer because decode is sequential. At mid depth, explain the KV cache, its linear growth, and how prompt caching reuses it. At senior depth, size a cache for a model layout, say why a filled 1M-token window is a memory problem before it is a compute problem, and map prefill, decode and memory onto the three cost levers. Expect "Why does the first token get slower with a longer prompt?" (prefill) and "Why do thinking tokens cost so much?" (they are decode).