Debugging an agent from a trace

Why it matters

An agent that gives a wrong answer has made five to fifty decisions on the way there, and the answer says nothing about which one went wrong. Engineers trained on deterministic systems read the final output, guess at the prompt, change a sentence, and watch the failure move rather than vanish. The trace exists so that you do not guess. This lesson is a procedure: symptom to trace to step to cause to fix to test. Interviewers ask "an agent gave a wrong answer, what do you do" at senior level because the answer separates people who have operated one from people who have demoed one.

The idea

Start from the symptom

Four symptoms cover most tickets: a wrong answer, a loop that never finished, a cost spike, and a slow run. Each says how to read the trace. A wrong answer is read forwards from the first step. A loop is read for repeated states. A cost spike is read with spans sorted by cost. A slow run is read with spans sorted by duration, looking at the gaps between spans as well, since a queue or a retry backoff shows up as a gap.

Find the trace

The request id on the ticket, or the user id and a time window, or a search over span attributes: stop reason equal to the step limit, cost above a threshold, a tool span with an error status. If finding the trace takes more than a minute, that is the first bug to fix.

Read the trajectory step by step

For each step, three questions. What did the model see: the exact messages in its context at that step, including the system prompt, the history, the tool results so far and the retrieved passages. What did it decide: the text or the tool call it emitted, with the arguments. What came back: the tool result or the next observation, as the model received it, not as the tool meant it. The first step where one of the three stops making sense is where you stop reading forwards and start reading closely.

In the example, the symptom is step 4, a confident wrong delivery date. The cause is step 3: the tool timed out, the handler returned the error as a plain string, and the model, told by the prompt to always answer, filled in a date from the user's own message. Editing the prompt would make it hedge. The fix is a handler that returns a typed error, a loop that retries once, and a refusal path when the retry fails.

step 4 input (diff against step 3)
+ tool_result lookup_order: "error: upstream timeout after 8000 ms"
  system: support-v14 ... "always give the customer a concrete answer"
step 4 output
  text: "Your order was delivered on Monday."    stop: end_turn

The usual culprits, in order

Check them in this order; the early ones are commoner and cheaper to confirm. Context missing or stale: the fact the model needed was never in its window, or an old version of it was. A tool result misread: the result was right and the model drew the wrong thing from it, usually because it was large, ambiguous or badly labelled. A tool error swallowed: an error returned as content, as in the example. A prompt version change: the version on the span is not the one you believe shipped. A loop with no new information: the same tool called with the same arguments, the same observation returned, and no step that changes the plan. A budget hit: the run stopped at a token, step or time limit and the last answer is whatever the model had at that moment.

Reproduce, fix, add to the eval set

Replay the trace: take the recorded context at the failing step and run the same prompt version against the same pinned model, with tool results served from the trace rather than live. If it fails the same way, you have a reproduction that costs one model call and no production traffic. Change one thing, the handler, the prompt or the retrieval, and replay again. When it passes, the recorded context and the expected behaviour become a case in the eval set, so the loop from the Building an eval set from production traces lesson closes and the release gate catches the regression next time.

What a good trace viewer shows

The tree with duration and cost on every span. The exact model input at every step, diffed against the previous step so you see what was added. Tool arguments and results side by side. The prompt version and resolved model id on every model span. A replay control on any step. Search over attributes. Anything less and the engineer reconstructs the context by hand, which is where the guessing comes back.

Never debug from the final answer alone

The final answer is the last line of a story. Reading it and editing the prompt is editing the ending without reading the plot. Open the trace first, every time, even when the cause looks obvious, because the obvious cause is the prompt and the real cause is usually a tool or the context.