Durable execution and checkpoints

Why it matters

An agent that answers in one call can live in a request handler. An agent that migrates a database, processes five hundred documents overnight, or waits for a manager's approval runs for hours or days, and in that time the service will be redeployed, the pod rescheduled, a model call will time out, and a human will go home. If the loop's state lives in a variable, each of those restarts the task from the beginning, repeating every paid call and side effect. Durable execution keeps the agent's state outside the process so a run can stop anywhere and continue from there. Interviewers raise it for any task longer than a request.

The idea

State outside the process

Everything the loop needs to continue is written somewhere that survives the process: the conversation, including tool calls and results; the plan and the current step; pending approvals and what they wait on; the pinned facts from compaction; the run's identity and version. A database row or a document per run is enough. The in-memory loop is a cache of that record, not the record itself.

Checkpoint after every lap

A lap is a model call plus the tool calls it made. After each lap, write the new state as a checkpoint: lap number, messages appended, tool results, the updated plan. The write lands before the next lap starts, so a crash loses at most one lap.

Idempotent tool calls

The interrupted lap may have called a tool before the process died; if the tool charged a card or sent an email, a naive replay repeats it. The Tool use fundamentals lessons gave the protection: every side-effecting call carries an idempotency key derived from the run id, the lap and the call, so the provider recognises the retry and returns the original result. Tool results are checkpointed too, so a replayed lap finds the result under that key and skips the call.

run=r_8f2a lap=301 tool=extract_invoice key=r_8f2a:301:extract_invoice:doc_4410 -> result cached, call skipped

Resuming

Resume loads the latest checkpoint, rebuilds the window from the stored messages, and starts the next lap. Any lap that began but did not checkpoint runs again, and with keyed tool calls that rerun is safe. A resume is routine: a deploy resumes every in-flight run by the same path.

Workflow engines

This is what durable workflow engines provide as a category: the loop is written as a workflow whose steps are persisted as they complete, with retries, timers and signals built in. A timer waits a day without a process holding the wait; a signal delivers an approval to a sleeping workflow. The engine replays recorded history to reconstruct state, so the workflow code must be deterministic: every model call and tool call goes through an activity whose result is recorded, never inline.

Pausing for a human

A human-in-the-loop step is a pause of unknown length. The agent checkpoints with a pending approval, notifies someone, and stops; no process waits. Days later the approval arrives as a signal and the run resumes with the decision in its window. Give every pause a timeout and an owner, and show pending runs in a queue a human can see.

Versioning the agent

A checkpoint written by version 12 of the agent may be resumed by version 14 after a deploy. If the message format, the tool names or the plan schema changed, the old checkpoint fails to load. Store the agent version with every checkpoint, keep readers for old formats or write a migration, and never rename a tool under running runs: add the new one, let in-flight runs drain, retire the old.

Observability of long runs

A run that spans days is watched like a service: active runs with their current step, time since last checkpoint, pending approvals and their age, retries per step, cost so far. Alert on a run that has not checkpointed in longer than a lap should take, and on spend past a per-run budget. The per-lap trace lives in the checkpoint store, so an operator can read the reasoning that led to step 300.