Streaming and latency

Why it matters

A model answer is not one event: it is a first token after a delay and then a stream of tokens at a steady rate, and the two halves have different causes, costs and fixes. Products that treat the answer as one blob wait for the last token before showing anything, and users call them slow even when the model is fast. Interviewers ask about latency because a chat system's feel is set here, and because "where did the other six seconds go" is the daily work of whoever runs one.

The idea

Two clocks

Time to first token (TTFT) is how long the user waits before anything appears; time to last token is when the full answer has arrived. For a 400-token chat answer, TTFT might be 600 milliseconds and the last token five seconds later; a voice agent needs a fraction of both.

Prefill and decode

Serving has two phases with different economics. Prefill processes the whole prompt in one parallel pass to build the key-value cache the model attends over; it is compute-bound, its time grows with input length, and nothing is emitted until it finishes, so a 30K-token prompt costs a visible fraction of a second before the first token. Decode produces one token per step, each step reading the whole cache; it is memory-bandwidth-bound, so the rate is roughly constant and total time grows with output length. TTFT is queue plus prefill; the rest is output tokens divided by the decode rate. The Serving lesson covers the caches and batching behind both.

What streaming changes

Streaming returns tokens as they are decoded. The request is the same with a flag set; the response becomes a sequence of events over a held-open connection, usually server-sent events or a chunked HTTP body, with content deltas and a final event carrying the stop reason and usage.

event: content_block_delta
data: {"type":"text_delta","text":"The order"}

event: content_block_delta
data: {"type":"text_delta","text":" shipped on"}

event: message_delta
data: {"stop_reason":"end_turn","usage":{"output_tokens":212}}

For the user, perceived latency collapses from time to last token to TTFT: text appears in under a second and arrives faster than most people read. For the server, three things get harder. Connections stay open for the length of the answer, so proxies and load balancers must not buffer and must tolerate long-lived responses. Structured output arrives as partial JSON, so parse incrementally or delay rendering until the block closes. And the user may leave mid-answer, so wire cancellation through: closing the stream stops decoding and stops the bill.

Perceived latency

Beyond streaming, two techniques move the feel without moving the model. Show progress stages while the model is silent ("searching", "reading three documents") so the gap is explained rather than blank. Render speculative UI, the layout, a skeleton, the citations panel, before the first token.

Budgets

Each product class has its own number. Chat: TTFT under a second and the answer streaming at a readable rate; past two seconds, users start to re-send. Voice: the whole turn, speech recognition plus model plus synthesis, under about 800 milliseconds before the pause reads as a stall, which forces a small model, a short prompt and streaming into the synthesiser. Batch: no interactive budget; a batch API that completes in minutes to hours at half price is the home for anything with no one waiting.

Where the seconds go

Retrieval sits before prefill: embedding the query, searching the index and reranking add a few hundred milliseconds in series. Each tool call is a full extra round trip: a new prefill over the grown prompt, a decode of the call, your handler's time, then another prefill and decode. An agent turn with three tool calls is four model calls in sequence plus three handlers, which is how a two-second model produces an eight-second answer. Rate-limit retries and a buffering proxy add more, invisibly.

Trade-offs

Streaming buys feel and costs engineering: every proxy, retry wrapper and client must understand a long-lived response, and a stream that dies halfway leaves a partial answer and no usage block. Structured output and streaming pull against each other; a schema is valid only when it closes. Progress stages and speculative UI can over-promise, showing "reading documents" while a tool call fails. Voice budgets rule out the model you would otherwise choose. And a batch API is cheap only if nobody is waiting; the moment a product depends on its results, its latency is yours.

In an interview

Separate the clocks first: "I measure time to first token and time to last token separately; the first is queue plus prefill and depends on input length, the second is output length over the decode rate. I stream so users see the first clock, not the second." At junior depth, know the two clocks and what streaming does for the user. At mid depth, explain prefill versus decode, the server-side cost of streaming, and a latency budget for chat and voice. At senior depth, decompose an eight-second answer into retrieval, serial tool calls, prefill and transport, and say what you would fix first. Expect "Why does a longer prompt slow the first token?" (prefill) and "How do you stream structured output?" (incremental parsing or render on close).