A voice turn is a chain of deadlines, not a request

The appointment agent worked on the laptop. A caller would say "yeah, Tuesday afternoon is fine", and the reply came back quickly enough that nobody in the demo room commented on it. Two weeks after launch the clinic's office manager forwarded a recording with a note that said "listen to what she does at eleven seconds". At eleven seconds the caller says "Tuesday afternoon is fine", and then there is silence, and at 1.9 seconds into that silence she says "hello?", and at 2.4 seconds — while the agent has finally begun to speak — she says "Tuesday, I said Tuesday", and now both parties are talking and neither one is listening.

Nothing in that call failed. No request errored, no timeout fired, no alert was raised. The pipeline's dashboard showed a mean end-to-end turn latency of 1.1 seconds against a target that had been written down as "about a second", and the team spent a week arguing about whether 1.9 seconds was even possible given that number, which it obviously was, because the recording existed. The argument was unresolvable because the team had one number for a process that has seven, and no way to attribute the extra 800 milliseconds to anything.

The system in that recording is a voice agent: a program that holds a spoken conversation with a person in real time, taking audio in from a microphone or a phone line, deciding what to say, and speaking back, with no keyboard or screen anywhere in the loop. This one answers the clinic's phone, asks which appointment the caller wants, and writes the change into the scheduling system. You have met its relatives when a bank's line offered to check a balance instead of transferring you, or when a support number asked what the problem was and acted on the answer rather than reading out a menu. Agents like it exist because speech is the only interface available to someone who is driving, holding a handset to their ear, or wearing gloves, and until recently nothing on the far end of a call could understand an open-ended sentence. Realtime is the load-bearing half of the name: the agent is holding a turn in a live conversation rather than transcribing a voicemail at leisure, so the conversation itself is what sets the deadline.

The clinic's agent is built as a chained pipeline, which means four services in a row. Automatic speech recognition — ASR, also called speech-to-text — turns the caller's audio into words, a turn detector decides the caller has stopped speaking, a language model reads the words and writes a reply, and text-to-speech renders that reply back into audio. Each stage waits for the one in front of it to produce something, so the delays add rather than overlap, and their total is the silence the caller sits in while nothing appears to be happening.

What the caller is actually measuring#

The interval a caller experiences begins at the moment they stop producing speech and ends at the moment sound reaches their ear. It is not the duration of any API call, and it does not begin when your server receives the last audio packet, because by then several tens of milliseconds have already been spent. Conversation analysts have measured the gap between turns across ten languages and found a median around 200 milliseconds, with the bulk of transitions landing under 500 milliseconds; the corpus work most often cited here is Stivers and colleagues' 2009 cross-linguistic study, and its practical consequence is that humans have a strong prior about how long a silence should last. Past roughly 800 milliseconds a listener starts to suspect the other party did not hear them. Past 1.5 seconds they act on that suspicion, which is what the caller in the recording did.

That prior is what makes voice different from a chat product. In chat, a spinner is a socially acceptable way to occupy 3 seconds of a user's attention, and a streamed first token at 600 milliseconds reads as fast. In a phone call there is no spinner; the only channel is audio, and the only thing silence communicates is that something is wrong. So the target is not "fast enough that the user does not get bored" — it is "inside the window where a human would have replied", and that window is fixed by everyone the caller has ever spoken to.

Seven stages, seven numbers#

The single number has to be broken into the stages that produce it, each with its own owner and its own failure mode. What follows is a budget for a chained pipeline — microphone to speech-to-text to model to speech synthesis to speaker — running from a browser over decent residential wifi. The numbers are the ones I would open a project with; they are targets to be measured against, not measurements. Each stage carries two of them, written as p50 and p95: the median, which half of turns come in under, and the 95th percentile, which all but the worst turn in twenty comes in under.

The two columns exist because they answer different questions. The p50 column tells you whether the design is viable at all; if the stages sum to 1,600 milliseconds at the median, no amount of operational care will save the product, and you need a different architecture. The p95 column tells you what fraction of turns will be bad, and it is where the recording from the clinic lives. Notice that the p95 numbers do not sum to a meaningful total, because the 95th percentile of a sum is not the sum of the 95th percentiles — stages fail independently, so the tail of the end-to-end distribution is dominated by whichever single stage happened to be slow on that turn. That is exactly why the aggregate number cannot be debugged: at p95 you are not looking at seven stages each 40% slower, you are looking at six normal stages and one that took 900 milliseconds.

A budget is also a product statement, and the same table is wrong for a different product. An outbound reminder call where the agent asks one closed question and the caller says "yes" tolerates almost nothing, because the exchange is short and every silence is a large fraction of it. A voice interface for a field technician reading a serial number off a panel tolerates a good deal more, since the human is doing something with their hands and expects the machine to be thinking. Support triage sits in between, and the useful move there is to set different targets for different turn types: acknowledgements and clarifying questions get the aggressive budget, and turns that require a lookup get a longer one with an explicit plan for filling the gap. Writing one number for every turn in the call is how teams end up optimizing the turns that were never the complaint.

The largest line in that table is turn detection — the decision that the caller has finished and it is the agent's turn to speak, also called endpointing — and it is the one most often left out of budgets entirely, because it does not look like work. Nothing is computing during those 300 milliseconds; the system is sitting through a hangover, the fixed stretch of continued silence it insists on before it will believe the caller has stopped. The waiting is charged to the caller's silence just the same, which means a team that shaves 80 milliseconds off model latency while leaving a 700-millisecond silence threshold in place has optimized the wrong tenth of the problem.

Measuring it means one clock and one identifier#

The instrument that makes the table real is a per-turn record with monotonic timestamps — readings from a clock that only ever moves forward and is never nudged by time synchronization — taken at the boundary of each stage, all on the same clock, keyed so that a turn can be found later. Wall-clock timestamps across machines will not do, since a 40-millisecond clock skew between your media server and your inference gateway is both common and larger than several of the stages you are trying to measure.

import time
from dataclasses import dataclass, field

@dataclass
class TurnTiming:
    call_id: str
    turn_index: int
    marks: dict[str, float] = field(default_factory=dict)

    def mark(self, name: str) -> None:
        self.marks[name] = time.monotonic()

    def stages(self) -> dict[str, float]:
        order = [
            "speech_end",        # last frame of user speech, as timestamped at capture
            "endpoint_decided",  # turn detector committed
            "transcript_final",
            "llm_first_token",
            "tts_first_chunk",
            "playback_start",    # client reports first sample rendered
        ]
        out, prev = {}, self.marks[order[0]]
        for name in order[1:]:
            out[name] = round((self.marks[name] - prev) * 1000, 1)
            prev = self.marks[name]
        out["total"] = round((self.marks[order[-1]] - self.marks[order[0]]) * 1000, 1)
        return out

Two of those marks are harder to get than the rest, and both are worth the trouble. speech_end has to be the capture-side timestamp of the last audio frame containing speech, not the moment your server decided speech had ended, because otherwise the endpointing hangover disappears from the measurement and the budget silently loses its biggest line. playback_start has to come from the client, since it is the only place that knows when the jitter buffer — the small reserve of audio a receiver holds back so that packets arriving unevenly still play smoothly — released the first sample; a server that logs "audio sent" is logging a number the caller never experienced.

A single turn from a staging run, printed from that structure, looks like this:

{
  "call_id": "c_9f3a", "turn_index": 4,
  "endpoint_decided": 712.0,
  "transcript_final": 96.4,
  "llm_first_token": 384.2,
  "tts_first_chunk": 163.8,
  "playback_start": 141.0,
  "total": 1497.4
}

Recording every turn this way is cheap — six floats and two identifiers, a few hundred bytes per turn — and the discipline that makes it useful is emitting the record even when the turn was fine. A pipeline that only logs slow turns cannot tell you what normal looks like this week, and cannot show you that the median crept from 1.0 to 1.25 seconds after a prompt change added 900 tokens of examples to every request. The comparison you want available at three in the morning is this call against the distribution, not this call against a target somebody wrote in a document eight months ago.

That trace answers the question the dashboard could not. The turn took 1.5 seconds, and 712 milliseconds of it was the system waiting to be certain the caller had stopped talking. Every other stage was within its p95 target. Halving the model's time to first token — the change the team had scoped as two weeks of work on prompt caching, where the provider stores the computed state of a prompt prefix so that an identical prefix on the next request does not have to be processed again — would have bought 190 milliseconds and left the turn at 1.3 seconds, still outside the window where the caller feels heard.

Where the budget framing stops being useful#

A latency budget is a claim about a distribution, and there are three things it does not do. It does not tell you whether the agent said anything worth waiting for, so a system that hits 800 milliseconds and produces confidently wrong answers will score well and fail in production; latency is a constraint you satisfy while optimizing something else. It does not cover the case where the delay is legitimate — a database lookup that genuinely takes 1.4 seconds is not a budget violation to be optimized away, it is a product situation that needs an answer in the audio channel, which is a separate design problem.

The third limitation is the one that catches teams later. A budget assumes the stages are independent lines you can move one at a time, and several of them are coupled by decisions made much earlier. Endpointing aggressiveness trades against how often the agent cuts people off. Jitter buffer depth trades directly against playback latency, so 60 milliseconds shaved off the client buffer arrives as audible dropouts on a lossy connection. Speaking faster gets the first syllable out sooner and makes the agent harder to interrupt cleanly. The budget tells you where the time is; it does not tell you which trades are acceptable in your product, and answering that requires knowing what each stage is actually doing.

There is also an assumption buried in the table that deserves naming. It is written for a chained pipeline, where transcription, inference and synthesis are three separate services with three separate first-response times. Direct speech-to-speech models collapse the middle three rows into one, which is a different budget with different failure modes and a different set of things you can no longer inspect. That comparison needs the rest of this material before it can be made honestly, so it comes last.

The number the budget cannot supply#

The 712-millisecond line in that trace is not a constant someone chose badly. It is the output of a decision — the caller has finished speaking — that the system makes on evidence that does not actually contain the answer. Lower the threshold and the agent begins talking over people who were pausing to think; raise it and every turn inherits the delay the clinic's caller reacted to. The rest of the budget is engineering, in the sense that faster hardware and better caching move it. This line is not, and the next chapter is about why the obvious way to compute it is measuring the wrong thing entirely.