Context engineering: what goes in the window

Why it matters

A 1M-token window does not mean you should fill it. Every token in the context costs money, adds latency and competes for the model's attention; past a point, adding material makes answers worse, not better. Prompt engineering is the wording of one message. Context engineering is the larger job: deciding, on every turn, which instructions, tools, facts, memories and history go into the window, in what form, and what stays out. It is the skill that separates systems that work in the demo from systems that still work on turn fifty, and it is where senior interviews now spend their time.

The idea

What competes for the window

Six kinds of content want in. Instructions: the system prompt. Tool definitions: name, description and schema for each tool, often several hundred tokens apiece. Retrieved facts: chunks pulled from a store for this turn. Memory: durable facts about the user or the task carried across sessions. Conversation history: prior turns, including tool calls and their results. Scratch state: the plan, intermediate results, what has been tried. Each is useful; together, unmanaged, they overflow or rot.

The budget view

Treat the window as a budget with three bands. Fixed overhead is paid every turn: system prompt plus tool definitions. Variable parts are filled per turn: retrieval, memory, history. Reserved output is the room the answer needs, subtracted up front; a model with no room left truncates its answer or its reasoning. A working budget for a 200K window might be 10K fixed, 120K variable under a hard ceiling, 32K reserved, and the rest as margin. Set the ceilings in code and enforce them; the model will not.

Compaction

History grows without bound and most of it stops mattering. Compaction replaces old turns with a summary: what was asked, what was decided, what remains open, after which the raw turns are dropped. Keep the most recent turns verbatim, because detail matters most there. Summaries lose things, so decisions and facts that must survive should also be written to a store outside the window. Raw tool results are the first candidates: a 40K-token log file read once has no business staying in the window for twenty more turns.

Progressive disclosure

Do not load everything in case it is needed. Thirty tool schemas cost 15K tokens a turn and make the model worse at choosing among them; load the five relevant to this task, or give it a search tool that returns schemas on demand. The same holds for documents: load a table of contents and let the model open sections, rather than the whole manual.

Context rot

Quality falls as the window fills with low-value tokens. Three mechanisms. Lost in the middle: attention favours the start and the end, so a fact buried at token 60,000 of 100,000 is often missed although it is present. Distraction: irrelevant chunks and stale tool output pull the model towards the wrong material. Dilution: with many instructions in play, each is followed less reliably. The consequence is that retrieval precision matters more than recall once the window is large; five right chunks beat fifty mixed ones.

Tool results are data

Anything that arrives through a tool, a retriever or an upload is content to reason about, not instruction to follow. Wrap it in delimiters, label its source, and keep the rules that govern it in the system prompt. The Prompt injection lesson treats the attack; the design rule is that the model can tell, from structure alone, which tokens carry authority.

	Stuff everything	Retrieve on demand	Summarise	Memory store
Cost per turn	Grows with the session; worst at the end	Bounded by top-k and chunk size	Bounded; one extra call per compaction	Small; a few facts per turn plus lookups
Fidelity	Highest until rot sets in	High for what was retrieved; blind to the rest	Loses detail; a wrong summary looks like a fact	High for what was written; stale if never updated
Latency	Prefill grows every turn	Plus a retrieval round trip	Plus a summarisation call now and then	Plus a store read; negligible
Complexity	None	An index, chunking, ranking to tune	Compaction policy, what to keep verbatim	Write policy, conflict rules, expiry

Trade-offs

Stuffing everything is simple and faithful for short tasks, then costs and rots. Retrieval keeps the window small and adds a retrieval system to tune and debug. Summaries cut cost and silently drop detail; a wrong summary is a stale fact that looks fresh. Memory stores persist facts and raise the hardest question: when a memory conflicts with what the user just said, which wins. Name the failure modes before they are named for you: stale facts from memory or an old summary; contradictory context, where two sources disagree and the model picks one at random; and lost in the middle, where the fact was there and was not read.

In an interview

Say: "I treat the window as a budget: fixed overhead, variable parts under a ceiling, reserved output. I compact history, disclose tools and documents on demand, and keep tool output as data." At junior depth, name the six kinds of content and the three bands. At mid depth, explain compaction, progressive disclosure, lost in the middle, and why precision beats recall. At senior depth, design the loop for a long-running agent: what survives compaction, how memory conflicts resolve, and the metrics that reveal rot, such as answer quality by turn number and tokens per turn. Expect: "The assistant gets worse after an hour of use; what do you check first?" Tokens per turn, then what is filling them.