Tracing LLM applications with OpenTelemetry

Why it matters

A model-backed request is not one call. It is a retrieval step, a model call that decides to use a tool, two tool calls and a second model call that writes the answer, spread across three services. When the answer is wrong, the question is which of those steps saw what. A log line that says "completed in 4.1 s" cannot answer it, and neither can the provider's dashboard, which sees only its own call. The Observability module in the System Design track covers logs, metrics and traces as a discipline; this lesson is about the fields only a model-backed system has, and about recording them without slowing the request or leaking user data. Interviewers ask because "how would you debug that" has no answer without it.

The idea

A trace is one request's story as a tree of spans, each a timed unit of work with a parent, a name and attributes. Standard tracing already gives you the HTTP span, the database span and the queue hop. A model-backed system adds span kinds with attributes nothing else in your stack has.

What to record per request

The model call. The model id as the provider resolved it, not the alias you asked for, since an alias can move under you. The prompt version. Inputs and outputs, or references to them in a store you control. Token counts in, out and cached, which are the cost ledger; the API surface lesson covers the usage object they come from. Latency to first token and to last. The stop reason, so a truncated or tool-calling response is visible without reading the body.

Tool calls. Name, arguments, result or a reference to it, duration and outcome. The arguments are what the model decided; the result is what it was then told. You need both to tell a bad decision from bad information.

Retrieval steps. The query as rewritten, the documents returned with ids and scores, and which of them made it into the prompt. Many "the model hallucinated" reports are retrieval that returned the wrong passage, and only this span shows it.

Reasoning transitions and state. In an agent, each loop iteration is a span: the step number, the decision the model emitted, any change to shared state. The Observability across agent boundaries lesson extends this to handoffs between agents.

Metadata on the root. User segment, prompt version, environment, experiment arm, tenant. These are the dimensions you will later slice cost and quality by; if they are not on the trace you cannot add them afterwards.

OpenTelemetry as the portable layer

OpenTelemetry gives you the span, the attribute, the context that carries a trace id across process boundaries, and an exporter to whatever backend you choose. Model-specific fields sit on ordinary spans under a gen_ai attribute namespace, so one tree shows the HTTP handler, the vector database query and the model call together. Instrument once, in your own thin client wrapper, and the data goes to any backend, model-specific or general. Instrument through a vendor SDK and you own that vendor's schema.

span   llm.call            trace 3c9e   parent 71b0   1.2 s   ok
gen_ai.request.model            mid-tier-2026-09
gen_ai.prompt.version           support-v14
gen_ai.usage.input_tokens       1900    cached 1400    output 60
gen_ai.response.finish_reason   tool_use
gen_ai.prompt                   ref: blob://prompts/3c9e-1  (sampled, redacted)
app.user_segment  enterprise    app.env  prod    app.experiment  rerank-b

The trace id travels in request headers between services and in the metadata field on the provider call where one exists, so a support ticket carrying a request id leads straight to the tree.

Privacy, cost and latency

Prompts carry user data. Store bodies by reference in a store with the data's own retention and access controls; redact or hash anything matching a PII pattern before it leaves the process; sample full bodies for a fraction of successful requests and keep them for every failure, gate hit and flagged answer. Metadata, token counts and timings are small and safe; keep those for every request.

Storage is the cost: an agent run with bodies attached can run to megabytes, so sample bodies and keep structure. Export asynchronously: spans are buffered in the process and shipped in batches by a background exporter, so a slow tracing backend never adds to the user's latency. A synchronous write to the trace store on the request path is the one way to make observability make the product worse.

Trade-offs

Every attribute trades a question you can later answer against bytes stored. Bodies by value make replay exact and make the trace store a PII store; by reference keeps the data where its policy lives and breaks replay the day the reference expires. Sampling keeps the bill down and loses the rare request you wanted, which is why failures are never sampled out. A vendor SDK is quick and locks the schema to that vendor; a wrapper of your own costs a day and survives a change. Spans without a prompt version are nearly useless a month later: the model and the prompt both change, and the trace cannot say which.

In an interview

Say: "Every request is a trace: a span per model call with model id, prompt version, tokens in, out and cached, latency and stop reason; a span per tool call with arguments and result; a span per retrieval with documents and scores; and the user segment and experiment on the root, exported asynchronously over OpenTelemetry." At junior depth, know what a trace and a span are and which fields a model call adds. At mid depth, describe the tree, the gen_ai attributes, body references and sampling. At senior depth, cover propagation across services, redaction before export, the retention split between metadata and bodies, and why instrumentation lives in your own wrapper. Expect: "Does tracing slow the request?" Not if the exporter is asynchronous and bodies are references. And: "Where do you put the prompt?" By reference, sampled, redacted.