Memory stores and recall

Why it matters

Where memory lives and how it comes back show up in every turn's cost and in every wrong answer. A fact stored where it cannot be found is forgotten; a fact recalled when irrelevant is noise; a fact recalled after the world has changed is a confident lie. Interviewers ask where the memory lives to see whether you treat memory as retrieval, with its ranking and freshness problems, or as a box that remembers.

The idea

Four shapes of store

A key-value store holds named facts: user.timezone, project.test_command. Writes are upserts by key, reads are exact lookups, and a fact has one current value, so revision is trivial. It has no notion of relevance.

A vector store holds free-text memories with embeddings. Reads are similarity searches: memories related to deploying the billing service. It finds things you did not know to ask for by name, and near-duplicates and stale versions just as readily.

A graph holds entities and relations: user works on project, project depends on service, service owned by team. Reads are traversals: everything within two hops of this ticket. It answers structural questions that similarity cannot, at the cost of an extraction step that decides what the entities are.

Plain files the agent edits: a notes file per project, a rules file it appends to. Reads are "load the file"; the model filters by reading. Cheapest to build, most transparent to users, first to outgrow the window.

Most systems combine two or three: key-value for identity and settings, vector for free-form memories, a file for procedural rules. A graph is added when the questions are about relationships.

Recall is retrieval

On each turn something decides what to pull. The query is the current user message plus the task state; the candidates are every memory in scope; the ranking combines relevance with recency and confidence, so a fact confirmed last week outranks a similar fact from last year. The budget is small on purpose: five to ten memories, under a thousand tokens, written into the window as one labelled block. Ten right facts beat fifty mixed ones.

In the system prompt or on demand

Stable, always-relevant memory goes in the system prompt: the user's name and role, standing preferences, the procedural rules. It loads once per session and benefits from prompt caching. Everything else is retrieved on demand, per turn, because loading it all costs more and dilutes what matters. The split moves: a fact recalled on most turns earns a place in the system prompt.

Memory services

Dedicated memory services now exist as a category: hosted layers that take a transcript, extract and store memories, resolve duplicates and return a ranked memory block per turn; Letta, Zep and Mem0 are names you will hear. They save building the pipeline; they also put personal data in another vendor's store and make the ranking a black box you evaluate rather than tune. Underneath, it is the same four shapes plus recall.

Memory is a cache of the world

A memory records what was true when it was written. The source of truth is elsewhere: the user's profile in your database, the deploy state in your deploy system, the file on disk. When both can be consulted, consult the source and keep memory for what the source cannot tell you, mostly preferences and history. When a memory is about something that has a source, store a pointer and a date, not a copy.

Cost of recall per turn

Recall is a retrieval round trip plus the tokens it adds. An embedding call and a vector query add tens of milliseconds; the memory block adds its tokens to every turn, so a thousand-token block over a hundred-turn session is a hundred thousand input tokens. Measure recall precision: of the memories recalled, how many did the answer use.

	Key-value	Vector	Graph	Files
Query shape	Exact lookup by name	Similarity to free text	Traversal: neighbours, paths	Load it all; the model filters
Freshness	One value per key; revision is an upsert	Old versions linger unless deleted by id	Relations go stale silently	Whatever the last edit left
Cost	Negligible	Embedding plus query per turn; storage grows	Extraction per write, traversal per read	Tokens per load; grows with the file
Use when	Identity facts and settings	Free-form notes and episodes	Questions about relationships	Procedural rules, small projects