Module 14 · Multi-agent systems · Lesson 5 of 5
Observability across agent boundaries
One trace id crosses every handoff, each agent and tool call is a span under the run, cost and latency fall out per agent, any agent can be replayed from its recorded input, and alerts fire on loops, budget hits and failed handoffs.
Why it matters
When a multi-agent run goes wrong, the question is never "what did the model say". It is "which of the six agents, on which hop, with what input, and how much did it cost before anyone noticed". A flat log answers none of that; you need one identifier that follows the request through every agent and tool, and a view of the run as a tree rather than a list. Interviewers ask because it is where the answer to "how would you debug this" either exists or does not. The Observability module in the Operating stage covers the tooling; this lesson is about what changes at agent boundaries.
The idea
One trace id for the whole run
Every run gets a trace id at the entry point. It goes into every handoff object, every tool call, every model request's metadata and every log line. Without it the run is six separate conversations that happen to be near each other in time. The handoff schema from the previous lesson carries it as a required field, so an agent cannot start work without it.
Spans per agent and per tool call
A trace is a tree of spans. The root is the run. Each agent invocation is a child span whose input is the handoff object and whose output is the structured result. Each model call inside the agent is a child of that, with tokens in and out, cost and latency; each tool call likewise, with arguments, result size, duration and outcome. The execution graph (which agent called which, in what order, with what) falls out of the parent ids; nobody reconstructs it from timestamps.
What the tree answers
Cost per agent. Sum the model-call spans under each agent span. The run cost $0.61 and the researcher was $0.44 of it, so that is where to look. Latency per hop. Each handoff is a span boundary with a start time; the gap between one result span ending and the next agent span starting is coordination overhead. Where it drifted. Walk the handoffs in order and read the goal field at each; the hop where the goal stopped matching the user's request is where the drift began. Replay. Because each agent span holds its full input and each model call holds its exact prompt, you can re-run one agent from its recorded input against a new prompt or model without re-running anything upstream, which is how you fix one agent in isolation.
Evaluate the thread, not the call
A multi-agent run is one conversation thread with the user, and the metric that matters is whether the thread achieved the goal within budget. Scoring each model call on its own tells you the writer wrote well from a wrong brief. Evaluate the root span: goal achieved, on budget, hops within range, no validation failures; then the agent-level checks under it, tool-call accuracy and agent goal accuracy per span. Sample traces from production into an eval set and the tree becomes the regression suite.
What to alert on
Alert on structure, not on sentiment. A hop count above the tail of good runs means a loop. A global or per-agent budget hit means a fan-out or a loop. A handoff validation failure means an agent produced the wrong shape, which is a deploy problem. A gap between hops above a threshold means coordination latency. These fire from span attributes, so they are cheap and name the agent.
Sampling and storage
A full trace of a fifty-call run with prompts and results attached runs to megabytes. Keep every span's metadata (ids, timing, tokens, cost, outcome) for every run; keep full prompt and result bodies for a sample of successful runs and for every run that hit a gate or failed validation. Prompts contain user data, so bodies get the retention policy the data would have anywhere else.
| Flat logs | Per-agent traces | End-to-end trace graph | |
|---|---|---|---|
| What you can answer | What each agent printed, in time order | What one agent did, with its cost and latency | Which agent, which hop, what input, what it cost, where it drifted |
| Cost | Cheapest to store, dearest to investigate | Moderate; one tree per agent | Highest storage; sample bodies, keep metadata |
| Setup | A logger | Instrument each agent runtime | Trace id through every handoff and tool, parent ids everywhere |
| Fails when | Two agents interleave; nothing links them | The bug is between agents, not inside one | An agent drops the trace id, which validation should catch |