Attention and the transformer

Why it matters

Every frontier model is a transformer, and every number that constrains your system follows from its shape: why prefill is fast and decode is slow, why the KV cache fills GPU memory. A design interview will ask why a 200K-token prompt is slow and expensive, and "because attention is quadratic" only counts as an answer if you can say what attention is.

The idea

A transformer is a stack of identical blocks, typically 30 to 120 of them in current models. Each block has two parts: an attention layer that lets every token gather information from other tokens, and a feed-forward layer that transforms each token's vector on its own. Between blocks runs the residual stream: one vector per token that each layer reads from and adds to, rather than replaces, so what early layers wrote survives to the end. At the top, a projection turns the last token's vector into a probability over the vocabulary: the next-token prediction.

What attention computes

Each token's vector is projected into three smaller vectors. A query says what this token is looking for; a key says what this token offers; a value is the content it hands over if chosen. For one token, attention scores its query against the key of every earlier token with a dot product, scales the scores and normalises them with a softmax so they sum to one, then adds the weighted sum of those tokens' values back to the residual stream. That is the whole mechanism: a soft lookup in which every token builds its own weighted view of the context.

One small intuition. In "The bank approved the loan because it was solvent", the token "it" must decide what it refers to. Its query, after training, resembles "a thing that can be solvent"; the key of "bank" matches far better than the key of "loan", so "bank" takes most of the weight and its value flows into "it". Nobody wrote that rule; it was learned because getting the referent right lowers loss.

Heads, position and the mask

Each layer runs several heads in parallel, each with its own smaller query, key and value projections, and concatenates their outputs. Different heads learn different relations: the previous token, the matching bracket, the subject of this verb.

Attention is order-blind, so position has to be injected. Current models rotate query and key vectors by an angle that grows with position (rotary embeddings), so each dot product depends on the relative distance between two tokens.

The causal mask sets the score for every later token to minus infinity before the softmax, so token number t can attend only to tokens one to t. In training the whole sequence is processed in one pass and every position predicts its successor without seeing it; at inference each new token sees everything before it.

Why the cost is quadratic

Every token scores against every earlier token, so n tokens cost on the order of n squared scores per head per layer: doubling the prompt quadruples the attention work in prefill. The keys and values of every token are also kept, in the KV cache, so decode does not recompute them; that cache grows linearly with context and is the main limit on how many requests fit on one GPU. Grouped-query attention, which shares one key and value set across several heads, and sliding-window layers are how 1M-token windows are made affordable; the Serving module covers paged attention and prefix caching.

Decoder-only

The original design paired an encoder with a decoder. Current models keep only the decoder: one causal stack that reads prompt and answer as a single sequence, simpler and easier to scale. The price is that the prompt is read under the causal mask too, so an early token never sees the question at the end, which is one reason instructions placed last often work better.

	Attention	Recurrence	Convolution
Training parallelism	Full: every position at once	Sequential, one step at a time	Full
Long-range dependencies	Any pair in one hop	Decay with distance through a fixed state	Reach grows only with depth
Cost per generated token	Grows with context; needs a KV cache	Constant; fixed-size state	Constant window
Memory at inference	Linear in context length	Fixed	Fixed
Where it lives now	Every frontier model	State-space hybrids for cheap long context	Vision and audio front-ends

Trade-offs

Quadratic attention buys exact recall of any earlier token at the price of compute that grows with the square of context, which is why long prompts cost more and wait longer. Long context is not free in quality either: facts in the middle of a long prompt are recalled less reliably, a trained bias that varies by model. State-space and linear-attention models keep a compressed state instead of a full cache and trade exact recall for cheap length; hybrids with a few full-attention layers are the current compromise.

In an interview

Say: "A transformer is stacked attention and feed-forward blocks. Attention lets every token take a weighted view of earlier tokens, and because that is all-pairs, cost is quadratic in context length and the KV cache grows with it." At junior depth, name the parts and the causal mask. At mid depth, explain query, key, value, heads, positional encoding and the quadratic cost. At senior depth, tie the cost to serving (prefill versus decode, KV cache memory, grouped-query attention, prefix caching) and to prompt design (instruction placement, lost in the middle). Expect: "Why is prefill fast and decode slow?" (prefill is one parallel pass; decode is one token per pass, bound by memory bandwidth). "What does the KV cache hold?" (every processed token's keys and values, per layer and head).