Module 12 · Agent types · Lesson 7 of 7
Voice and real-time agents
Sub-second spoken turns: cascaded pipeline against speech-native model, the latency budget, barge-in, tool calls mid-speech, fillers, telephony constraints, and evaluation by task completion, latency percentiles and transcription errors.
Why it matters
A voice agent is a text agent with a clock attached. The user stops speaking and expects sound back within about a second; silence beyond that reads as broken, and talking over the user reads as rude. It must fit a budget measured in hundreds of milliseconds and survive noisy audio, interruptions and a phone line. Interviewers ask about voice to see whether you can carry a budget through a pipeline and say what happens when a tool call takes four seconds while someone waits on the line.
The idea
Two architectures
The cascaded pipeline chains three models: speech to text, a language model, text to speech. Each is replaceable, each produces text you can log and test, and the language model is whatever you already use, tools and all. The cost is latency at each hop and the loss of everything that is not words: tone, hesitation, overlap. A speech-native model takes audio in and produces audio out in one model, with lower latency and a natural sense of timing and prosody, at the cost of control: fewer model choices, harder to inject retrieved text or enforce a script, and tool use still runs through text underneath.
The latency budget
Count from the moment the user stops speaking. Endpointing, deciding they have stopped, takes 200 to 400 ms, because a pause is ambiguous. Transcribing the final words takes under 200 ms when recognition has been streaming during the speech. The model's first token takes 300 to 600 ms with a short prompt and a fast model. Synthesising the first sentence and starting playback takes 100 to 300 ms with streaming synthesis. The target is first audio in under about a second; anything over 1.5 seconds needs a filler. Every stage streams into the next: the model receives partial transcripts, the synthesiser receives the model's output sentence by sentence, and audio plays while the model is still writing.
Interruptions and turn-taking
People interrupt. Barge-in means detecting the user's voice during playback, stopping the audio within about 100 ms, cancelling the model's generation, and recording in the transcript what was actually heard, not what was about to be said. Without it the agent talks over the user; done badly, background noise cuts the agent off mid-sentence. Turn-taking runs the other way too: the agent must not answer a pause that was the user thinking, which is why endpointing uses both silence and the shape of the transcript.
Tool calls during speech
A lookup takes longer than the budget. The agent starts speaking before the result arrives: an acknowledgement ("let me check that booking"), then the answer when the tool returns, with the tool call streamed as soon as the model decides to make it rather than after the whole turn. Fillers are a strategy, not a hack: a short, varied acknowledgement covers a few seconds; beyond that the agent says what it is doing and roughly how long it will take. Silence while a tool runs is the single most common voice failure.
Telephony and in-car
A phone line is narrowband audio with noise, codecs and dropped packets, which raises transcription errors on names, numbers and addresses, the exact things a booking depends on. The fix is read-back confirmation for anything that matters, and spelling for names. In a car the microphone competes with road noise, the user cannot look at a screen, and a wrong action (navigating, calling) is hard to undo mid-drive, so confirmations are spoken and short.
Evaluation
Task completion per call, graded from the transcript and the tool log. Latency percentiles, p50 and p95, for time to first audio and for barge-in stop time; the p95 is the number users remember. Transcription errors: word error rate on your own audio, and specifically on names and numbers.
| Cascaded pipeline | Speech-native model | |
|---|---|---|
| Latency | Sum of three hops; under a second with streaming at every stage | Lower; one model, no transcription hop |
| Control | Full: swap any stage, log text, inject retrieval, enforce scripts | Limited: fewer models, harder to constrain what is said |
| Cost | Three services, each priced; the language model dominates | One service, usually priced per audio minute; often higher per turn |
| Quality | Loses tone, overlap and hesitation; exact text for tools | Natural prosody and timing; weaker on exact names and numbers without read-back |