A median cannot show you the eight percent complaining

The dashboard read 1.05 seconds median turn latency against a 1.2-second budget, and the support queue had 47 tickets that month describing "long pauses", against roughly 590 calls. The two facts were both true and the meeting spent forty minutes trying to make one of them wrong. The engineering position was that the budget was met, so the complaints must be about something else — tone of voice, the agent asking too many questions, users unfamiliar with talking to a machine. The proposed action was to add a filler phrase to every turn so the silence would feel occupied.

That conclusion is available only because the aggregate is the wrong shape for the question. A median is the middle of a distribution and carries no information about its right tail, and complaints come exclusively from the right tail. Worse, the metric was per turn while the complaint is per call, and those differ by more than most people's intuition allows.

The arithmetic that makes 4% into 39%#

Suppose 4% of turns exceed 2.5 seconds. That is a small number and it will not move a median at all. A support call in this product averages 12 turns, and if bad turns were independent, the probability that a call contains at least one is one minus 0.96 to the twelfth power, which is 39%. Even with a considerably more forgiving threshold — say a caller only complains after two bad turns in a call — the per-call rate stays in the double digits.

The direction of that arithmetic is what matters. Per-turn metrics systematically understate how many people had a bad experience, because experience is per call, and the longer your calls the wider the gap. So the first change is not instrumentation at all; it is the definition of the headline number. Something like the fraction of calls containing at least one turn over 2 seconds, reported next to the median, describes the complaint in the same units the complaint arrives in. On the clinic's data that number was 11%, which is finally in the same neighbourhood as the 8% of calls that generated tickets.

Making a bad turn findable#

The per-turn record from chapter 1 already has the stage boundaries. What turns it into an investigative tool is the identifiers and attributes attached to it, because the question you will actually ask is never "what is the p99" but "what happened on the call this ticket refers to, and what do the calls like it have in common". That requires the turn record to carry enough context to define "calls like it" after the fact. The code below writes those records as OpenTelemetry spans, a span being one timed unit of work — here, one voice turn — carrying a start, an end and a set of key-value attributes you can filter and group by later.

from opentelemetry import trace
tracer = trace.get_tracer("voice")

def emit_turn(t: TurnTiming, ctx: dict) -> None:
    stages = t.stages()
    with tracer.start_as_current_span("voice.turn") as span:
        span.set_attribute("call.id", t.call_id)
        span.set_attribute("turn.index", t.turn_index)
        # stage durations, in ms
        for name, ms in stages.items():
            span.set_attribute(f"stage.{name}_ms", ms)
        # the attributes that let you slice later
        span.set_attribute("transport", ctx["transport"])        # webrtc | ws | pstn
        span.set_attribute("carrier", ctx.get("carrier", ""))
        span.set_attribute("region", ctx["region"])
        span.set_attribute("model", ctx["model"])
        span.set_attribute("prompt_tokens", ctx["prompt_tokens"])
        span.set_attribute("cache_hit", ctx["cache_hit"])
        span.set_attribute("tool_calls", ctx["tool_calls"])
        span.set_attribute("endpoint_ms_used", ctx["endpoint_ms_used"])
        span.set_attribute("interrupted", ctx["interrupted"])

Three of those attributes exist purely because of earlier chapters. The transport and carrier fields separate populations whose floors are different, so a PSTN call and a browser call never land in the same percentile bucket. The endpoint_ms_used field records which threshold the conditioning logic actually chose for that turn, which is the difference between knowing that turn detection took 1,100 milliseconds and knowing that it took 1,100 milliseconds because the caller was reading digits and the policy asked for it. And interrupted marks turns that went through cancellation, which have a legitimately different shape and will otherwise pollute every percentile you compute.

One measurement in that record cannot be taken on the server at all, and leaving it out is the most common gap in voice telemetry. The quantity the caller experienced is the silence between their last syllable and the first sound of the reply, and both endpoints of that interval are on their device. A server that times from "last audio frame received" to "first audio frame sent" is missing the uplink transit at the start and the downlink plus jitter buffer at the end, which on a mobile connection is 150 to 300 milliseconds of the very thing being complained about. The client has to report its own two marks — playback start, and the capture timestamp of the last speech frame — and the turn record has to carry both alongside the server-side stages, so that the sum of the parts can be checked against the whole.

Sampling deserves a decision rather than a default. Trace sampling at 1% is standard practice for high-volume web services and it is wrong here: at 590 calls a month you would capture six of them, and the ones you need are specifically the rare ones. Voice traffic is low volume by web standards and turn records are a few hundred bytes, so keep all of them, and keep the call identifier somewhere a support agent can paste it — on the ticket, in the CRM record, next to the recording.

Joining a complaint to a call#

The cohort analysis below is only possible if the tickets can be turned into a list of call identifiers, and this is where most instrumentation efforts stall — not on the telemetry, which is straightforward, but on the fact that the complaint arrives as prose from an office manager three days later with no identifier attached. Fixing that is a product decision made early. Every call record needs a stable identifier that appears in whatever system the human uses: the recording filename, the activity logged against the patient in the CRM — the customer relationship management system, the tool the front desk actually lives in — the row the receptionist sees when she looks up the patient. When a complaint is "the call with Mrs. Alvarez on Tuesday afternoon", the lookup has to be one query on patient and date, and the output has to be a call identifier that joins directly to the turn records.

Failing that, there is a weaker fallback that is still much better than nothing: infer the cohort. Calls where the caller repeated themselves, said "hello?" mid-call, or where the agent's turn was followed by immediate overlapping speech are all detectable from transcripts, and they correlate strongly with the calls people complain about. Building a nightly job that flags such calls gives you a synthetic complaining cohort that is available on every call rather than on the 8% where someone bothered to write in, and it lets you watch the number move after a change instead of waiting a month for tickets.

The same identifier makes alerting possible in the shape that matches the problem. An alert on median turn latency will never fire on this failure, because the median was fine throughout. An alert on the fraction of calls containing a turn over 2 seconds, evaluated hourly against the previous week's value for the same hour, fires on the day the regression ships.

Pulling the cohort rather than the aggregate#

With the records stored, the investigation is a query rather than a hunt. Compare the stage distributions of the complaining cohort against everything else, and let the difference identify the stage:

SELECT
  CASE WHEN c.has_complaint THEN 'complaining' ELSE 'baseline' END AS cohort,
  count(*)                                                          AS turns,
  round(percentile_cont(0.50) WITHIN GROUP (ORDER BY endpoint_ms))   AS endpoint_p50,
  round(percentile_cont(0.95) WITHIN GROUP (ORDER BY endpoint_ms))   AS endpoint_p95,
  round(percentile_cont(0.50) WITHIN GROUP (ORDER BY llm_ttft_ms))   AS ttft_p50,
  round(percentile_cont(0.95) WITHIN GROUP (ORDER BY llm_ttft_ms))   AS ttft_p95,
  round(percentile_cont(0.95) WITHIN GROUP (ORDER BY tts_first_ms))  AS tts_p95,
  round(percentile_cont(0.95) WITHIN GROUP (ORDER BY playback_ms))   AS play_p95
FROM turn_timing t
JOIN call c USING (call_id)
WHERE t.interrupted = false AND t.created_at > now() - interval '30 days'
GROUP BY 1;

On the clinic's data the result had one column that differed and five that did not:

Model time to first token in the complaining cohort had a median three and a half times the baseline median. That is not a tail effect within those calls; it is a different distribution, which means the complaining calls were a distinguishable population rather than unlucky draws. The prompt_tokens attribute finished the diagnosis: time to first token rose with the turn index, because the conversation history grew, and it rose sharply past a threshold because prompt caching stopped covering the prefix once each call's history diverged.

Long calls were the complaining calls. The filler phrase proposed in the meeting would have played on every turn of every call, including the 89% that were fine, and would have covered nothing on the turns that were not — a 1.8-second wait is not concealed by a 700-millisecond "let me see". The actual fixes were ordinary: summarize history beyond turn eight rather than resending it, restructure the prompt so the cacheable prefix stays stable across turns, and drop the recognizer's alternates from the context — the runner-up transcriptions it returns alongside its best guess — which were contributing about 40% of the token growth for no measured benefit.

The limits of stage attribution#

Per-stage tracing answers where the time went and refuses to answer two other questions people bring to it. It does not tell you whether the reply was any good, so a system with a beautiful latency profile that misroutes a third of callers will look healthy in every chart here; conversational quality needs recordings, transcripts and human review, and the two investigations rarely share a dashboard.

It also does not attribute delay that is not in a stage. If your media server's event loop is blocked for 400 milliseconds by a synchronous call, that time is inside whichever stage boundary it happened to fall between, and the trace will implicate that stage falsely. The check is to compare the sum of stage durations against the independently measured wall-clock silence experienced by the client, and to investigate any call where they disagree by more than a few tens of milliseconds — a discrepancy there means the delay is in your process rather than in any component you are timing.

The third limit is that the tail is not always a bug. The complaining cohort in this investigation had a real regression behind it, and the next investigation will find a stage whose 1.4 seconds are the honest duration of a database query that has to happen. That case cannot be fixed by making the stage faster, and covering it up is not as simple as the meeting assumed.