Context windows and the token budget

Why it matters

A million-token window makes "put everything in" the path of least resistance, and it is the most expensive mistake in applied LLM work. Every token in the window is paid for on every call, slows the first token, and competes for attention with the tokens that matter. Interviewers ask "what goes in the prompt, and what do you do when it does not fit" because the answer separates people who have run a chat product from people who have read about one.

The idea

The window is the model's entire working memory for one call: everything it knows about your task, beyond training, is in there or nowhere.

What the window holds

Left to right, a production prompt is the system prompt, the tool definitions, the conversation history, retrieved chunks for this turn, any memory the application injects about the user, and the space the answer will occupy. The first two are the same on every call; the middle three change per request; the last is a reservation that comes out of the same window: a model with a 1M-token window and a 128K max output cannot take a 950K-token prompt and write 100K tokens.

Why 1M does not mean "everything"

Four reasons, in the order an interviewer wants them. Cost: at a mid-tier price of a few dollars per million input tokens, a 500K-token prompt costs over a dollar per call before output, and a chat product makes thousands of calls an hour. Latency: time to first token grows with prompt length, because prefill must process every input token before decoding starts. Attention dilution: the more unrelated material in the window, the more attention spreads across it, and precision on the relevant part drops. Position: models recall the start and end of a long prompt better than the middle, the lost-in-the-middle effect, so a fact at token 400,000 is less likely to be used than the same fact at token 2,000. A needle-in-a-haystack pass shows a model can find one string, not reason over everything at once.

Budgeting

Budget the window like memory on a constrained device: fixed overhead, a variable region, a reservation.

window                1,000,000
system + tools            6,000   fixed, cached
reserved output          16,000   never given to input
retrieved chunks         12,000   8 chunks x 1,500
memory                    2,000
history                 <= 40,000  trim oldest turns past this
working budget           76,000   of 1,000,000 available

A working budget far below the window is normal; the gap is headroom.

Count, do not estimate. Use the model's own tokeniser or the provider's count endpoint before sending; character estimates miss by a third on code and by more on non-English text. Alert when a request crosses 80 percent of its budget.

Trim history from the oldest turn, in whole user-assistant pairs, never leaving a tool call without its result; the system prompt and the most recent turns stay intact.

Summarise older turns into a rolling summary when the conversation is long and early context still matters: a cheap extra call every few turns, and a summary block replacing the turns it covers.

Retrieve instead of inlining: a 300-page manual belongs in an index, and the eight chunks that match this question belong in the prompt. The Retrieval lessons cover how.

Prompt caching

The fixed prefix is where the money goes, and it never changes. Providers cache the processed state of a prefix and charge cache reads at 2.5 to 10 percent of the base input price. Two rules make it work: put the stable material first (system prompt, tools, reference documents) and the variable material last, so the prefix bytes are identical call to call; and keep anything dynamic, a timestamp, a user name, a request id, out of the prefix, because one changed byte invalidates everything after it. With a 50K-token prefix and 2K tokens of variable content, caching turns most of the input bill into a rounding error.

Trade-offs

Trimming is simple and forgets. Summarising keeps the gist, loses detail, and adds a call whose failures surface only when a user asks about something the summary dropped. Retrieval scales to any corpus and adds a whole system to tune and evaluate. Long context is the easiest to build and the dearest to run, and it degrades quietly rather than failing loudly. Caching cuts the bill but demands discipline: an engineer who adds the current date to the top of the system prompt takes the hit rate to zero, and nobody notices until the invoice arrives. And the budget is model-specific: tokenisers differ, so a prompt that fits one model may not fit another.

In an interview

Say it in one breath: "I budget the window like memory: a fixed prefix that I cache, a variable section that I trim or retrieve, and a reserved output region that the input is never allowed to eat." At junior depth, name the components and explain why a bigger window is not free. At mid depth, show a budget with numbers, the trimming rules, and the caching prefix rules. At senior depth, discuss attention dilution and position effects, the choice among trim, summarise and retrieve for a given product, and how you would measure cache hit rate and cost per conversation. Expect "Why not just use the full million?" (cost, latency and recall), and "What breaks the cache?" (any change to the prefix bytes).