Parallel calls, errors and retries

Why it matters

One tool call is easy. A production agent makes hundreds, some at once, against services that are slow, flaky or billed per call. The failures that reach the on-call engineer are rarely "the model said something wrong". They are "the agent sent the invoice twice", "the run cost forty dollars looping on a search that returned nothing", and "the tool hung and the user waited ninety seconds for an error". This is distributed-systems hygiene applied to a client that happens to be a language model, and interviewers probe it because it is where a demo turns into a product.

The idea

Several calls in one turn

A model can emit several tool-use blocks in one response: weather in Paris, Rome and Oslo, each with its own call id. The runtime should run independent calls concurrently and return every result in one message, one tool result per id, in any order, because the ids do the matching. Running them one after another wastes wall-clock time; returning them in separate messages wastes model round trips. Dependent calls cannot be parallel, and update_event and delete_event on the same id in one turn is a race; the runtime can detect conflicting writes and serialise them.

Idempotency

A retry is only safe when doing the thing twice equals doing it once. Reads qualify. Writes do not: a send_email that times out after the mail server accepted the message, then gets retried, sends two emails. The fix is the one payment APIs use, an idempotency key: a unique id the caller attaches so the server recognises the repeat and returns the original outcome instead of acting again. The call id the model generated serves within a turn; across runs, derive the key from the arguments and a run id. Say in the description which tools have side effects, so the model does not casually re-issue them either.

Timeouts and partial results

Every call gets a deadline. When it fires, return a result that says so rather than hanging the turn: timed out after ten seconds, the query may still be running, try a narrower window. If a batch of three has two successes and one timeout, return all three; the model can judge whether the partial picture is enough. Partial results beat none, as long as they are labelled.

tool_result id=call_7 ok     {"city":"Paris","temp_c":11}
tool_result id=call_8 ok     {"city":"Rome","temp_c":17}
tool_result id=call_9 error  {"kind":"timeout","after_ms":10000,"hint":"retry with a smaller window"}

Return the error, or abort

An error has two possible audiences. Return it to the model when the model can plausibly fix it: a validation failure, a not-found, a timeout, a rate limit with a retry-after. Give it three parts: what happened, why, and what to try next. Abort the run instead when the model cannot help: credentials revoked, a spend budget exhausted, a policy check failed. Showing the model those invites creative workarounds; the Excessive agency lesson explains why that is a security finding.

Budgets, limits and the trace

A failing tool and an eager model make an infinite loop with a bill attached. Set a step limit per run, a count of model turns after which the runtime stops and reports. Set a retry budget per tool: three attempts, then a hard error that says to stop retrying. Detect repeats: the same tool with the same arguments twice in a row is a signal to return "that call already failed; change the approach" instead of running it again. Rate-limit the tools themselves, because the model has no sense of a quota; a search tool capped at twenty calls a minute is a fuse. And log every call: tool name, arguments, duration, outcome, bytes returned, under the run id. The Observability lesson builds on that trace; without it nobody can answer "what did the agent do".

Trade-offs

Parallel execution needs a runtime that notices when calls are not safe to run together. Idempotency keys need server support, or a dedupe store you maintain yourself. Short timeouts cut latency but turn slow successes into reported failures; long ones keep users waiting. Rich errors cost tokens and a turn each time, and a model free to recover will try six variations on a hopeless call before a step limit stops it. Logging arguments records whatever the user typed, so the trace is now sensitive data with a retention question.

In an interview

Say it in one line: "Independent calls run concurrently and join into one result message; writes carry idempotency keys; every call has a timeout, a retry budget and a log line; errors the model can fix go back to it, the rest abort the run." At junior depth, know that a turn can carry several calls and that each result matches by id. At mid depth, explain idempotency keys, timeouts with partial results, and the three-part error. At senior depth, discuss step limits, repeat detection, per-tool rate limits and what the trace must capture. Expect the follow-ups: "How do you stop a retried payment charging twice?" (an idempotency key the server honours) and "What does the model see when one of three parallel calls fails?" (all three results, one marked as an error with a hint).