The context window as short-term memory

Why it matters

A language model has no memory of its own. Each call starts from nothing, reads the window and writes an answer; whatever is not in the window did not happen. So the window is the agent's short-term memory, and you decide what it holds. In an agent that runs forty laps, every tool result stays unless you remove it, the window swells, cost climbs, and the model starts missing things that are present. Interviewers ask because the demo works and the long session does not.

The idea

What short-term memory holds

Treat the window as working memory with four residents. Instructions are fixed. History is the sequence of user turns, model turns and tool calls. Tool results are what the tools returned: a file, a search result, a log. Scratch state is the model's own notes: the plan, what was tried, what remains. The Context engineering lesson budgets these; this lesson follows them through time.

Growth per lap

One lap is one model call plus the tool calls it makes. Nothing leaves the window on its own, so each lap adds the model's output and every tool result. A coding agent that reads a 30K-token file at lap 2 and a 40K diff at lap 5 carries 70K tokens of tool output into every later lap, needed or not. By lap 20 the cost per lap has grown tenfold, latency with it, and what matters now sits among what mattered once.

Context rot

Quality falls as the window fills with low-value tokens: lost in the middle, distraction, dilution. The symptom is an agent that was precise for ten laps and vague at thirty, or that re-reads a file it already read. Rot sets in long before the hard limit; a window 60 per cent full of stale tool output is worse than one 20 per cent full of the right material.

Compaction

Compaction shrinks history on purpose. Summarise the older turns into a few hundred tokens: what was asked, done and decided, and what is open. Keep the most recent turns verbatim, because detail matters most where the work is. And keep a pinned list outside the summary: the user's constraints, decisions made, identifiers and paths discovered, as short bullet lines copied forward unchanged on every compaction. The summary is lossy by design; the pinned list is not.

Dropping tool output after use

A tool result is usually consumed within a lap or two. Once the model has acted on a file read or a search, replace the raw result with a reference: the path, the query, a one-line gist, the lap it was read at. If the model needs it again it can call the tool again, which is cheaper than carrying 40K tokens for twenty laps.

[lap 5] read src/billing/invoice.ts (1,240 lines) -> dropped after use; re-read if needed

The compaction trap

The summariser is a model call, and it decides what matters. A constraint stated once in turn 3 ("do not touch the payments module") is exactly what a summary compresses to "the user gave some guidance". The defence is structural: constraints and decisions go to the pinned list the moment they appear, by rule rather than by the summariser's judgement, and the summariser is told to list open constraints explicitly.

When to compact

Two triggers. A token threshold, say 70 per cent of the working ceiling, checked after each lap. And a phase boundary: compact before a long sub-task so it runs in a clean window, and after it ends, fold its laps into one paragraph of outcome.

Measuring the effect

Log tokens per lap, the number of compactions, and task success by lap number. A policy that halves tokens per lap but drops success at lap 30 from 85 to 60 per cent has summarised something it needed. Keep an evaluation set of long tasks, because short tasks never exercise this code.

Trade-offs

Keeping everything is faithful and simple, and right for short sessions; at length it rots and costs. Truncating the oldest turns is cheap and crude: it discards the original request first. Summarising bounds cost at the price of fidelity, and a wrong summary looks like a fact. Pinned facts plus a summary is the usual production shape; its cost is a write policy that decides what gets pinned, and a pin goes stale when a decision is reversed and the old line is not removed.

In an interview

Say: "The window is the agent's short-term memory: tool output becomes a reference once used, old turns are summarised at a token threshold or a phase boundary, recent turns stay verbatim, and constraints live in a pinned list that survives every compaction." At junior depth, know that nothing leaves the window by itself and that cost grows per lap. At mid depth, describe compaction, dropping tool output and the pinned list. At senior depth, discuss the compaction trap, phase-aware triggers and measuring success by lap number. Expect: "The agent forgot a rule the user set at the start; what happened?" It was summarised away; it should have been pinned.