Speaker labels are inferred, except when they are recorded

A sales-coaching product computed talk ratio from diarised transcripts and reported that one rep spoke for 38% of her calls, which put her in the top decile for listening and earned a mention in a quarterly business review deck. Diarisation is the step that made that number possible: given a recording with several voices in it, decide which stretches of audio belong to which speaker, without being told in advance who they are or even how many of them there are. A transcript that has been through it carries a speaker label against every segment, which is what lets a coaching tool add up one person's seconds. Someone eventually listened to a call. She had spoken for about 61% of it. The diarisation had assigned most of her speech and most of the customer's to the same cluster on calls where both were women with similar pitch, then split the remainder by acoustic condition rather than by person, so the label boundaries tracked the customer's move from a car to an office rather than the change of speaker.

The failure is more interesting than a bad model, because the same product had a second bug that pointed at the same root cause. Its live meeting view showed captions attributed to Speaker 1 and Speaker 2, and about forty seconds into a call the labels would sometimes swap for everything already on screen. Users reported it as a rendering glitch. It was the diarisation working exactly as designed.

Clustering is a decision about the whole recording#

A conventional diarisation pipeline does five things, and only the first two are causal.

Voice activity detection and segmentation look at local audio. Embedding extraction looks at one segment and returns a speaker embedding, a fixed-length vector — 192 numbers in the ECAPA-TDNN case above — produced by a network trained so that two segments of the same person's voice land close together in that space and two different people land far apart, regardless of what either of them said. Clustering looks at every segment in the recording at once, because it has to: the labels are not names, they are cluster identities, and a cluster only exists relative to the other clusters. Deciding whether segment 3 belongs with segment 47 requires having segment 47.

The number of speakers is usually unknown, and it is inferred from the same global view — either by cutting the agglomerative dendrogram — the tree recording the order in which the closest pairs of segments were merged — at a chosen distance threshold, or by looking for the eigengap, the largest jump between consecutive eigenvalues of the matrix of pairwise segment similarities, whose position is taken as the number of groups the data supports. That estimate is unstable while audio is short. Forty seconds into a two-party call, the embeddings from a single speaker recorded through two acoustic conditions can look as far apart as two speakers, so the estimate lands on three; when the fourth minute supplies more evidence, the estimate drops to two and every previous assignment is recomputed. Cluster labels also carry no inherent order, so even a re-run that partitions identically may name the same partition differently, which is why the swap in the meeting UI applied retroactively to text already on screen.

This is what makes offline diarisation genuinely good and live diarisation genuinely hard. Online variants exist — incremental clustering with a frozen prefix, or end-to-end neural diarisation trained to emit per-frame speaker activity with a fixed maximum speaker count — and they work, at a cost of roughly five to ten percentage points of diarisation error compared with the offline version of the same system on the same audio. The design choice they force is stark: either you allow labels to be revised, and your UI and downstream consumers must handle a message that changes who said it, or you freeze each label at emission and accept that the early ones are drawn from a distribution the system had not yet learned.

If a downstream agent acts on speaker identity in real time — routing a compliance disclosure to the rep, suppressing the agent's own words from the caller's transcript, triggering on "the customer asked for a refund" — that action is being taken on a value that may be rewritten. The safe pattern is the one from the previous chapter applied to a different field: treat the live label as provisional, defer anything irreversible to the point where the recording is complete and can be diarised offline, and reconcile.

There is a structural limit in the clustering formulation that no amount of tuning removes. Each segment gets one embedding and therefore one label, so a stretch of audio where two people speak simultaneously can only be assigned to one of them. Overlap is 8% to 15% of speech time in two-party phone calls and higher in meetings, which puts a ceiling on what a clustering pipeline can score before any other error is considered. The research direction that addresses it directly is end-to-end neural diarisation, which predicts per-frame activity for each speaker independently — several speakers can be active in the same frame — trained with permutation-invariant loss so the arbitrary ordering of output slots does not penalise a correct partition. Those systems handle overlap properly and struggle with speaker counts beyond the small maximum they were trained for, so the current production compromise is usually a hybrid: neural diarisation over local windows, clustering to stitch the windows together, which inherits the global dependency described above along with the improved overlap handling.

Timestamps have the same character#

Word-level timings look like measurements and are not. A transcription model produces them in one of two ways, and both are inferences.

The first is that the decoder emits timestamp tokens as part of its output vocabulary, so the model is predicting when the same way it predicts what — a language-modelling operation conditioned on audio, subject to the same pressures as any other token, including the tendency to produce plausible round numbers and to drift after long silences. The second is dynamic time warping over the decoder's cross-attention weights — the numbers recording how strongly each output word attended to each frame of audio as it was produced — which aligns output tokens to input frames by finding a monotonic path through that matrix. That is better grounded, and it is still an alignment inferred from where the model attended rather than a measurement of when energy appeared in the signal.

The practical consequence is drift that grows within a decoding window. On a 30-second window with continuous speech, timings from attention-based alignment are typically good to a couple of hundred milliseconds; after a long pause, a filled hesitation, or a stretch of music the model partially hallucinated through, errors of 500ms to well over a second are ordinary. Forced alignment fixes it: run a frame-level CTC phoneme model over the audio with the transcript already known — CTC, connectionist temporal classification, being a training scheme that has the network label every 10 or 20ms frame with a phoneme, the smallest unit of speech sound, or with a blank — and solve for the alignment directly, which brings word boundaries to roughly ±30 to ±50ms. This is the technique WhisperX packages, pairing a transcription model with a separate alignment pass and a diarisation pass over the same audio.

Whether that precision matters is a product question with a sharp answer in at least one case. Redacting a spoken card number by muting a span depends on the span being right, and a 400ms error at the start of the span leaves the first digit audible while a 400ms error at the end leaves the last two.

{
  "words": [
    {"word": "card",   "start": 12.10, "end": 12.38, "confidence": 0.99},
    {"word": "number", "start": 12.38, "end": 12.79, "confidence": 0.99},
    {"word": "is",     "start": 12.79, "end": 12.88, "confidence": 0.98},
    {"word": "four",   "start": 13.02, "end": 13.24, "confidence": 0.91},
    {"word": "five",   "start": 13.24, "end": 13.49, "confidence": 0.88},
    {"word": "one",    "start": 13.49, "end": 13.71, "confidence": 0.94},
    {"word": "two",    "start": 13.71, "end": 13.96, "confidence": 0.90}
  ]
}

# Mute the PAN (primary account number) span with a margin that reflects
# the alignment's real error, not the two decimal places the JSON prints.
MARGIN=0.25
ffmpeg -i call_88214.wav \
  -af "volume=enable='between(t,$(echo "13.02-$MARGIN" | bc),$(echo "13.96+$MARGIN" | bc))':volume=0" \
  redacted_88214.wav

Two decimal places in a JSON field imply a precision the pipeline does not have. Either run forced alignment and shrink the margin, or keep the margin and accept that you are muting a quarter-second of surrounding speech. A cheap independent check is to compare each word boundary against the nearest energy transition from a voice activity detector run separately over the audio; systematic offsets in one direction indicate the decoder's alignment is lagging, which is common at the end of long segments.

When the answer is recorded rather than inferred#

All of the above is the situation where you have one mixed audio stream and must recover speaker structure from it. A large fraction of production audio is not that. A SIP provider recording a two-party call can write the caller to the left channel and the agent to the right, because at the moment of recording the two legs are still separate RTP streams — RTP being the protocol that actually carries the audio packets of each participant's connection — and their membership is therefore known rather than deduced. Conferencing platforms expose per-participant audio tracks for the same reason. Where those exist, the speaker attribution problem is not a modelling problem.

# Split the stereo recording into two mono files, then transcribe each alone.
ffmpeg -i call_88214_stereo.wav \
  -map_channel 0.0.0 caller.wav \
  -map_channel 0.0.1 agent.wav

# Transcribe per channel; merge by word start time. No clustering anywhere.
caller = asr.transcribe("caller.wav", timestamps="word")
agent  = asr.transcribe("agent.wav",  timestamps="word")

merged = sorted(
    [{"speaker": "caller", **w} for w in caller.words]
    + [{"speaker": "agent", **w} for w in agent.words],
    key=lambda w: w["start"],
)

Two properties of this deserve emphasis. Overlapped speech stops being a problem, because when both parties talk at once each channel still contains one voice and both get transcribed; single-channel diarisation must either pick a winner or produce a segment that belongs to neither. And the attribution is exact rather than 89% or 94% accurate, which removes an entire class of downstream uncertainty.

The failure mode to watch for is bleed. Echo cancellation is imperfect, and the agent's voice frequently appears in the caller's channel at −18 to −25dB, which is quiet to a human and entirely transcribable by a recogniser. The result is phantom turns: the agent's sentence appearing twice, once on each channel, a few tens of milliseconds apart. Gate on relative energy — for each transcribed span, compare RMS in that window across the two channels and drop the span from the channel where it is 12dB or more quieter — or deduplicate by comparing overlapping spans textually. Also resist the temptation to enable diarisation on top of multichannel transcription, which several APIs will happily let you do; it re-introduces clustering error on audio that already has the answer.

The decision follows from what the recording gives you, and Deepgram's guidance on choosing between multichannel and diarisation makes the same point: use channels when you have them, diarise when you do not, and do not run both.

What diarisation is not#

It is not speaker recognition. Diarisation answers "how many voices are here and which segments share one", and it does so with arbitrary labels that carry no identity and do not persist across recordings — cluster 0 in Monday's call has no relationship to cluster 0 in Tuesday's. Attaching a name requires enrolment, a stored voiceprint, and a verification step, which is a biometric system with consent requirements, retention policy and regulatory weight that diarisation does not carry. Products conflate the two constantly, usually at the point where someone asks to see "all the calls where Dr Okafor spoke".

It is also not voice activity detection, though it contains one. And a mixed conference-room recording from a single table microphone is the case where none of this chapter's escape hatches apply: there is one channel, several speakers, heavy overlap and reverberation, and the only structural improvements available are physical, from a microphone array running beamforming — combining several microphones with small time offsets so the array listens more strongly in one direction than another — to per-participant streams captured by the conferencing client.

That leaves a question this chapter has been assuming an answer to. The table above quotes 88–96% attribution accuracy, and both of the product failures at the start would have shown up as a number if anyone had been computing one. The next chapter is about what that number is, what it counts, and how much it moves depending on how you choose to count.