The augmented LLM: retrieval, tools, memory

Why it matters

Every agent pattern, every workflow and every RAG system is built from the same unit: a model with three things attached. Retrieval gives it facts it was not trained on. Tools let it act on the world. Memory lets it carry state across turns and sessions. The guide that named the workflow-versus-agent distinction calls this unit the augmented LLM and treats it as the building block of everything else. Interviewers like it because a candidate who can draw the unit can explain any system by saying which augmentations it uses and how they share one context window.

The idea

One mechanism, three augmentations

The model has exactly one input, the context window, and exactly one way to ask for anything, a tool call. Every augmentation arrives through those two doors. Retrieval is text the runtime puts into the context before the model speaks, or that the model fetches with a search tool mid-turn. Tools are functions the model can request by name, whose results come back as context. Memory is a store the runtime reads at the start of a turn and writes at the end, or that the model reads and writes through tools of its own. The model does not know the difference. From the inside, retrieval and memory are both text that appeared, and a tool is a tool.

One turn, three augmentations composed

Take a support assistant handling "my last order never arrived". The runtime assembles the context: the system prompt, the tool definitions, a memory read saying this customer has complained twice before, and retrieved policy chunks on delivery claims. The model calls get_order; the runtime executes it and appends the result. The model drafts a reply and calls remember with a note that a claim was opened today, which the runtime writes to the store. One turn, three augmentations, each through the same context-and-tools mechanism, each ending up as tokens.

The budget they share

Retrieval chunks, tool definitions, tool results and memory all land in the same context window, so they compete for the same tokens on every call. Twenty tools at 150 tokens each cost 3,000 tokens before anything else. Five retrieved chunks at 400 tokens are 2,000 more. A dump of the customer's history can be anything. The ceiling is the model's window, but the working budget is far smaller: cost rises with every token on every lap, and models attend less reliably to a crowded context. The runtime's first job is deciding what earns its place; the Context engineering lesson is that decision in detail.

The runtime's job

The runtime is the code around the model, and it has three duties on every turn. Assemble the context: pick the memory to read, the chunks to retrieve, the tools to expose, and order them. Execute the calls: validate arguments, apply permissions, run the function, bound the time, format the result. Persist the state: write memory, log the trace, store whatever the next turn or the next session needs. Frameworks differ in how they hide those duties, but when something goes wrong it is almost always inside one of the three.

Why this is the reusable unit

Prompt chaining is several augmented LLMs in a row. Routing is one that picks which of the others runs next. Orchestrator-workers is one that spawns several. An agent is one placed inside a loop. Every pattern in the Agents stage is a way of arranging copies of this unit, and every agent type, from a coding agent to a research agent, differs only in which tools, which retrieval and which memory the unit is given. Learn the unit and the patterns stop looking like separate inventions.

	Retrieval	Tools	Memory
What it adds	Facts the model was not trained on	Actions and live data	State across turns and sessions
Latency	One search before or during the turn	One round trip per call, plus the function itself	A read at the start, a write at the end
Failure mode	Wrong chunks answered with confidence	Failed, slow or misleading results built upon	Stale or poisoned facts that persist
Where it is taught	The Retrieval stage	The Tool use modules	The Memory and state module