Every term between the user's last word and the first syllable

The escalation said the voice agent felt sluggish and the fix was a faster language model. Someone had benchmarked two models offline, found one with a time to first token of 180 ms against the incumbent's 400 ms, and written a migration plan. The migration would have cost three weeks of prompt and eval rework and bought 220 ms out of a turn that took 1,580.

Nobody in the room could say what the other 1,180 ms was doing, which is the normal state of affairs. Voice pipelines are assembled from components owned by different teams and instrumented by their vendors, each of which reports its own stage honestly and none of which reports the gap between stages. The first useful move is not an optimisation; it is a list of every term between the moment a user stops speaking and the moment they hear a syllable, with a number next to each.

The six terms#

Measured on the dispatch agent, over 4,000 turns from the production region:

Two of those terms are constants because you chose them. Endpointing is a threshold in a config file, and the playout buffer is the prebuffer target from the first chapter — both are policy rather than performance, and both show a p95 identical to their p50 for that reason. The other four are distributions, and their p95s are two to three times their medians, which is the shape of anything that involves a queue.

The terms are not all serial in a well-built pipeline, which matters for how you read the total. ASR finalisation overlaps the moment the endpoint fires, because the recogniser has been maintaining a hypothesis all along. Synthesis of the first sentence overlaps generation of the rest of the answer, which is the chunking from the last chapter. And a speculative pipeline can start prompt processing before the endpoint fires at all.

Against that list, the model swap moves the third term from 400 ms to 180 ms and leaves the other five alone. It is a 14% improvement in the total, purchased with three weeks and a full re-evaluation of behaviour, and it is not the largest available win by a wide margin.

The largest term is the one nobody wants to touch#

Endpointing is 500 ms, which is 32% of the turn, and it is a constant you can change by editing a number. That is why it is the first thing people cut and the first thing that gets reverted.

Endpointing is not the same thing as voice activity detection, though they are often implemented in the same component and confused constantly. Voice activity detection answers whether the current 20 ms frame contains speech. Endpointing answers whether the user has finished their turn, which is a decision about intent, and the standard implementation approximates it with "VAD has reported no speech for N milliseconds". The approximation is the problem, because silence inside a turn and silence at the end of a turn look identical to a frame-level detector.

People pause mid-sentence constantly, and they pause longest exactly where the information is. On a sample of 1,200 turns from the dispatch agent's support queue, 18% contained an internal silence of at least 300 ms, and among turns where the user was reading out an identifier — "the load number is... four seven... two two one" — that rose to 41%. Cutting the threshold from 500 ms to 200 ms buys 300 ms on every turn and interrupts the caller on roughly one turn in six, disproportionately on the turns carrying the data you actually needed. The user experience of being cut off mid-identifier is far worse than 300 ms of extra wait, and it compounds, since the recovery turn costs a full extra round trip.

There are three ways to buy some of that 300 ms back without paying for it in interruptions.

Semantic endpointing makes the threshold conditional on the transcript. A small classifier — a few million parameters, or a fine-tuned encoder running in under 15 ms on CPU — reads the current partial and predicts whether it is a plausible complete turn. "The load number is four seven two two one" is complete; "the load number is four seven" is not. You then hold 160 ms of silence when the classifier says complete and 900 ms when it says incomplete. On a distribution where 70% of turns end syntactically complete, the mean threshold falls to about 380 ms while the interruption rate drops below the fixed-500 ms baseline, because the mid-identifier pauses are precisely the ones the classifier flags as incomplete.

Speculative execution overlaps the wait instead of shortening it. At 200 ms of silence, send the current partial transcript to the language model and start prefill, the pass that processes the whole prompt before any output token is sampled; if speech resumes, cancel the request and discard the work. When the endpoint does fire at 500 ms, the prompt is already processed and you are 300 ms into generation. The cost is tokens: with a 900-token prompt and a 22% cancellation rate, you are paying about 200 extra prompt tokens per turn, which under prompt caching is close to free and without it is a real line item. The cost you have to watch more carefully is that a speculative generation started on a truncated transcript will produce an answer to the wrong question if you forget to invalidate it when the transcript changes.

Filler audio covers the gap rather than closing it. Playing a 300 ms "Mm-hm" or "Let me check" while the real answer is generated makes the turn feel responsive and adds nothing to the information rate; it also commits you to a stance before you know the answer, which reads badly when the real reply turns out to be a refusal. Use it for tool-calling turns where you already know the answer will take two seconds, and not as a general-purpose latency patch.

Budget the tail, because the mean is not what gets remembered#

A pipeline whose p50 is 1,580 ms and whose p95 is 3,400 ms is experienced as a system that mostly works and sometimes hangs, and users describe systems like that as slow. Averages hide this completely: shaving 100 ms off the median while the tail grows produces a dashboard that improves and a product that gets worse.

The tempting way to compute a tail budget is to add up the p95 of every term, which gives 3,290 ms here. That number is wrong in both directions and should not be used. It overstates the tail when the terms are independent, because it assumes every stage has its bad day on the same turn, and independent terms rarely align — the actual measured p95 of the end-to-end value was 2,410 ms, well under the sum. It understates the tail when the terms are correlated, which happens exactly when you care: a degraded region, a noisy neighbour on the GPU pool, or a network path that is slow for the recogniser and the synthesiser at once. Correlation means the p99 can exceed the naive sum. Measure the distribution of the end-to-end quantity directly, and use the per-term percentiles only to attribute it.

Attribution is the part that makes tail work tractable. Emit one span per term on every turn — a span being a single timed, named interval recorded against a trace, which is the unit every tracing backend indexes — keyed by turn id, and query the slowest decile for which term dominated it.

from opentelemetry import trace
tracer = trace.get_tracer("voice.turn")

def run_turn(audio_stream, ctx):
    with tracer.start_as_current_span("turn") as turn:
        turn.set_attribute("session.id", ctx.session_id)
        with tracer.start_as_current_span("endpointing") as s:
            transcript, endpoint_ms = wait_for_endpoint(audio_stream)
            s.set_attribute("threshold_ms", ctx.endpoint_threshold_ms)
            s.set_attribute("speculative_cancelled", ctx.spec_cancelled)
        with tracer.start_as_current_span("asr_final"):
            final = finalise(transcript)
        with tracer.start_as_current_span("llm_first_token") as s:
            first_token, stream = generate(final, ctx)
            s.set_attribute("prompt_tokens", ctx.prompt_tokens)
            s.set_attribute("cache_hit", ctx.cache_hit)
        with tracer.start_as_current_span("tts_first_audio"):
            chunk = synthesise_first_chunk(first_token, stream)
        with tracer.start_as_current_span("playout") as s:
            ttfa = sink.play(chunk)
            s.set_attribute("prebuffer_ms", ctx.prebuffer_ms)
        turn.set_attribute("ttfa_ms", ttfa * 1000)

-- which term owns the slow turns?
WITH slow AS (
  SELECT turn_id FROM turns
  WHERE ttfa_ms > (SELECT approx_percentile(ttfa_ms, 0.90) FROM turns)
    AND ts > now() - interval '24 hours'
)
SELECT span_name,
       count(*)                        AS turns,
       approx_percentile(dur_ms, 0.50) AS p50_in_slow_turns,
       sum(dur_ms) * 1.0 / (SELECT sum(dur_ms) FROM spans
                            WHERE turn_id IN (SELECT turn_id FROM slow))
                                       AS share_of_slow_time
FROM spans WHERE turn_id IN (SELECT turn_id FROM slow)
GROUP BY span_name ORDER BY share_of_slow_time DESC;

Run on the dispatch agent, that query put 46% of the slow decile's time in llm_first_token and 24% in tts_first_audio, with endpointing's fixed 500 ms shrinking to 15% of a slow turn simply because it does not grow. That is the real argument for the model swap, and it is an argument about the tail rather than the median: the incumbent model's p95 first token was 1,300 ms against a 400 ms median, and the candidate's was 240 ms against 180 ms. The win was never the 220 ms of median. It was a variance reduction of roughly a second on one turn in twenty.

What this budget does not cover#

The six terms measure the wait before the first syllable and say nothing about the rest of the turn. A reply that starts at 1,580 ms and then stalls at second fourteen because generation runs below realtime scores identically on this budget and terribly with users, which is why the buffer arithmetic from the first chapter is a separate measurement rather than a term in this table.

It also excludes barge-in, which has its own budget running the other direction. When a user starts speaking over the agent, the time from their first phoneme to the agent falling silent should be under 200 ms, and it decomposes into VAD detection, a signal to the playback sink, and however much audio is already buffered downstream — which means a large playout buffer, and any buffering inside the client's audio stack, is latency you cannot cancel. Teams that fix stutter by raising the prebuffer to 800 ms sometimes discover they have made interruption feel broken, since up to 800 ms of already-committed audio keeps playing after the user has started talking.

Finally, none of this touches connection setup. An 80 ms network term assumes warm, pooled connections to the recogniser and the synthesiser. A pipeline that opens a fresh TLS connection per turn pays a handshake of 100–200 ms twice, and it will not show up in any vendor's latency dashboard because the vendor starts counting when the request arrives.

Latency work of this kind assumes the voice coming out the other end is one you are entitled to use. That is a separate question with its own failure mode, and it arrives the first time someone asks whether the agent can sound like a specific real person.