Long context versus retrieval

Why it matters

When frontier windows reached a million tokens, a claim followed: retrieval is dead, put the documents in the prompt. It did not happen, for reasons that are arithmetic and product constraints rather than model quality. A million tokens is about 750,000 English words, roughly 1,500 pages, which is a lot of documents and also not very many. Interviewers now ask the question in its real form: given this corpus, this traffic and this budget, do you retrieve, do you stuff, or both? The strong answer works the cost per query, names the quality effects of a full window, and knows the constraints that retrieval satisfies and a prompt cannot.

The idea

What a big window buys

A whole corpus in the prompt removes the retrieval system: no chunking, no index, no embeddings, no recall failures, no freshness pipeline. The model sees everything and can join facts across documents without anyone designing a graph. For a small corpus that rarely changes, read by one analyst for one afternoon, this is the right design, and building a retriever for it is over-engineering.

The arithmetic

Take a 200-page handbook, about 150,000 tokens, and a mid-lineup list price of $3 per million input tokens. Every question that carries the handbook costs about $0.45 in input before the answer is written. Prompt caching changes that: cache reads cost 2.5 to 10 percent of the base input price, so a cached reread of the same 150,000 tokens is roughly $0.01 to $0.045. Retrieval instead sends the question plus five chunks, perhaps 4,000 tokens, for about $0.012, plus the cost of running an index. At the top of the cache range the two are close; at the bottom the cache wins on price, as long as the prefix is identical on every call and the cache stays warm. Change one line of the handbook and the next read is full price again. Multiply by a thousand questions a day and the design decision is a line in the budget.

What the full window does to quality

Two effects are worth naming. The needle in a haystack test hides one fact in a long context and asks for it; current models pass it well, which is why the claim sounded plausible. The lost in the middle effect is subtler: when many facts compete, those near the start and the end of the context are used more reliably than those in the middle, and irrelevant material dilutes attention. Finding one needle is easy; reasoning over twenty needles scattered through 150,000 tokens of hay is where answers degrade. The Context engineering lesson covers the same effect from the prompt side.

Where retrieval still wins

Four constraints decide it. Size and change: a hundred thousand documents do not fit, and a corpus that changes hourly invalidates the cache as fast as it is built. Cost and latency per query: prefill on 150,000 tokens takes seconds before the first output token, every time, and uncached it is the dearest thing in the system. Precision and citations: a retriever returns the five passages the answer rests on, with their locations, which is what a user clicks and an evaluator checks; a full-window answer cites "the handbook". Access control: a prompt is all or nothing, so a user who may read only part of the corpus cannot be served from a shared prefix. The Permission-aware retrieval lesson builds on that point.

The hybrid

The strongest current designs do both. Retrieve, but retrieve large: a few whole sections or documents, 20,000 to 50,000 tokens, rather than fifty 500-token chunks. The model gets coherent text with its surroundings, the retriever still provides filtering, citations and access control, and the window is far from full, so the middle is not lost.

	Long context	Retrieval	Retrieve, then long context
Cost per query	Every token, every time; cache cuts it to 2.5 to 10% if the prefix holds	A few thousand tokens plus index cost	Tens of thousands of tokens; no index-wide cache
Latency	Seconds of prefill	Retrieval round trip, short prefill	Retrieval plus a medium prefill
Freshness	Re-send or re-cache the corpus	Re-embed the changed chunk	Re-embed the changed section
Citations	Weak; the whole corpus	Exact passages	Sections, with passages inside them
Access control	All or nothing per prefix	Filtered in the index	Filtered in the index