Failure modes: cascades, loops, runaway cost

Why it matters

One agent fails one way at a time. Several agents fail together: the bad output of one becomes the confident input of the next, two agents volley a task forever, a fan-out spends a week's budget in an afternoon, and nobody can say whose fault it was. The controls are mostly cheap and mostly boring, and a senior engineer is expected to list them before the first incident rather than after it.

The idea

Cascading errors

Agent A invents a customer id. Agent B looks it up, finds nothing, and helpfully constructs a plausible record. Agent C emails the constructed customer. Each agent did something locally reasonable with poisoned input, and the cascade was possible because every handoff was trusted. The control is validation at every boundary: shape checks (is this the schema) and semantic checks (does this id exist).

Delegation loops

The supervisor asks the specialist. The specialist judges the task out of scope and hands it back. The supervisor, seeing a fresh task, delegates it again. The control is a hop counter in the handoff that each agent increments, with a limit that aborts the run; a per-agent step limit; and loop detection, meaning the same task hash arriving twice at the same agent returns an error instead of running.

Runaway cost

A researcher fans out to ten sub-searches, each sub-search fans out to five page readers, and each reader calls a summariser. Fifty agents and three hundred model calls from one question, with no single place that sees the total. The control is a global budget for the run (tokens, money, wall-clock) that the runtime decrements on every call, per-agent budgets so one branch cannot eat the rest, and a fan-out limit per level.

Inconsistent results

Two agents answer the same question from different data: one reads the live database, one reads yesterday's export, and the report prints both numbers. The control is one source per fact, resolved before the fan-out; a single writer for shared state; and a reconciliation step that compares overlapping outputs and flags disagreement rather than concatenating.

Deadlock on shared state

Agent A waits for a field that B will write; B waits for a field that A will write. Both time out, or neither does. The control is a single writer per key, a dependency order fixed by the orchestrator rather than discovered at run time, and a timeout on every wait.

Duplicated side effects

A supervisor retries a specialist that timed out after it had already sent the email. Two parallel agents both decide the ticket needs a reply. The control is an idempotency key on every write, derived from the run id and the action, and a rule that side-effecting tools live in one agent only.

The blame problem

The output is wrong and four agents touched it. Without per-agent attribution the fix is a guess. The control is per-agent evals, so each agent's contract is tested on its own inputs in CI, and a trace that records every handoff object and every result, which the next lesson covers.

The control set

Lay the controls out as gates. Between agents: handoff validation, a hop counter and a loop check. Around each agent: a step limit, a token budget, a retry budget, and a circuit breaker that stops calling an agent whose last few calls failed. Around the run: a global budget, a deadline, and a kill switch, a flag a human can flip that halts every agent in the run at its next step. On shared state: a single writer per key. On side effects: idempotency keys and one owning agent. Every gate is enforced by the runtime, not by prompts; a prompt that says "do not loop" is a wish.

	Cascade	Delegation loop	Runaway cost	Duplicated side effect
Symptom	Confident wrong output downstream of one bad hop	The same task bouncing between two agents	One run costs ten to a hundred times the median	The customer gets two emails or two refunds
Control	Validate every handoff, shape and meaning	Hop counter, loop detection, step limit	Global and per-agent budgets, fan-out limit	Idempotency keys; one agent owns side effects
Enforced where	At the boundary, in the runtime	In the handoff and the per-agent loop	In the call path, before each model call	In the tool layer
Detected by	Validation failures in the trace	Hop-count alert	Budget-hit alert	Duplicate-key log line