Diarisation error rate sums three failures against reference speech time

A meeting-notes product sends a summary email after every call, and the summary includes an action-item list with a name against each line. In one week's sample, three of those lines named the wrong person, and one of the three was Marcus will send the revised SOW by Friday — the statement of work, the document that defines what a consultancy has agreed to deliver — when Priya had said it. Marcus did not send the SOW. The engineer who investigated reported a diarisation error rate of 11% on the product's meeting corpus, and the PM's response was that 11% sounded acceptable and the launch should proceed. Both the number and the reaction to it deserve more scrutiny than they got, in opposite directions.

Three failures over one denominator#

Diarisation error rate is defined against a reference that is exactly as unglamorous as the hand transcripts from chapter two: intervals of time, each marked with a start, a duration and a speaker label, produced by a person listening to the audio. The standard interchange format is RTTM — rich transcription time marked, a whitespace-separated format inherited from the NIST evaluations — with one line per speech interval and most of its fields unused for this purpose.

SPEAKER meeting_0412 1 12.480 3.220 <NA> <NA> priya <NA> <NA>
SPEAKER meeting_0412 1 15.910 0.640 <NA> <NA> marcus <NA> <NA>
SPEAKER meeting_0412 1 16.400 5.180 <NA> <NA> priya <NA> <NA>
SPEAKER meeting_0412 1 21.700 8.940 <NA> <NA> dana <NA> <NA>

Producing that reference is the step teams assume is out of reach, and it is not. An annotator working in Audacity or ELAN with the waveform on screen marks speaker regions at roughly three to five times realtime, and the effort drops to about one and a half times realtime when they correct a machine diarisation instead of marking from scratch — the same bootstrap-and-correct approach, with the same anchoring hazard, as the transcription references. Eleven meetings averaging just under ten minutes of speech each is a couple of days of work for one person, and it is the only way any statement about speaker attribution in this product becomes checkable. Boundary marks are the noisiest part of the annotation; two annotators over the same file will place the same turn boundary within about 120ms of each other most of the time and disagree by more on soft onsets, which matters directly for the scoring decision below.

Given a reference like that and a hypothesis in the same format, scoring walks the timeline and accumulates three quantities. Missed speech is reference time where somebody was speaking and the system said nobody was. False alarm is time where the system found a speaker and the reference has silence, which in practice means keyboard noise, door slams, laughter and the ventilation. Speaker confusion is time where both agree somebody spoke and they disagree about who. Their sum is divided by the total duration of speech in the reference:

DER = (missed + false_alarm + confusion) / total_reference_speech

Because hypothesis labels are arbitrary cluster identities, scoring first solves a one-to-one mapping between hypothesis clusters and reference speakers that minimises the resulting error — the Hungarian algorithm on a cost matrix of shared time. A system that partitions the audio perfectly but calls Priya "speaker_2" is not penalised, which is the right behaviour and also the reason you cannot read the labels themselves as meaning anything until that mapping has been applied.

On the meeting corpus the components came out like this, over 1,842 seconds of reference speech across eleven meetings:

The split is worth reading before the total. Confusion dominating means the voice activity detection is broadly right and the clustering is what fails, which points at embedding quality, speaker similarity, or a wrong speaker-count estimate — a different set of remedies from a missed-speech-dominated profile, where the fix is usually the VAD's threshold or an aggressive noise gate upstream.

from pyannote.core import Annotation
from pyannote.metrics.diarization import DiarizationErrorRate

metric = DiarizationErrorRate(collar=0.25, skip_overlap=True)
for ref, hyp in corpus:                      # both pyannote Annotation objects
    metric(ref, hyp)

report = metric.report(display=False)
print(float(metric))                          # 0.110
print(metric["missed detection"],             # 61.0
      metric["false alarm"],                  # 44.4
      metric["confusion"])                    # 97.9

The number moves with how you decide to count#

Two arguments in that constructor carry more weight than anything about the model. A collar excludes a window around every reference boundary from scoring, on the reasoning that a human annotator's own boundary marks are only good to somewhere between 100 and 200 milliseconds, so charging a system for a 150ms disagreement measures annotation noise. The NIST convention is 250ms on each side of every boundary, which forgives a surprising amount of audio in a conversation with frequent turn-taking. skip_overlap excludes regions where the reference has two or more simultaneous speakers, which are exactly the regions a clustering pipeline cannot get right by construction.

Scoring the same eleven meetings four ways, with the same hypothesis files, gives four numbers that would each be reported as "our DER":

The system did not change. DIHARD, in the last row, is a series of open research challenges that deliberately scores hard audio with nothing forgiven, which is why its published numbers look so much worse than a vendor's for systems of comparable quality; a vendor quoting 11% and a challenge evaluation quoting 26% can be describing identical output, and the gap between the two is entirely a scoring policy. Whenever a diarisation number appears in a comparison, the collar and the overlap policy belong beside it, in the same way the normaliser belongs beside a WER.

For this product the fourth row is closer to the truth that matters, because the audio is a single mixed channel from a laptop microphone in a room with four people, and the overlap it excludes is real speech that real users expect in their notes. Chapter five's escape hatch does not apply: there is no separate channel per speaker, so the forgiving score has nothing behind it. Meetings joined through a conferencing platform that exposes per-participant tracks are a different product with a different ceiling, and moving the ingestion path there is a larger accuracy win than any model change discussed in this chapter.

What the product actually needs to be true#

None of the four numbers answers the PM's question, because the product does not attribute seconds, it attributes sentences, and specifically it attributes the twelve or so sentences per meeting that become action items. Doing for speakers what chapter three did for words means measuring the thing the feature depends on.

Where confusion time lands turns out to be structured. Of the 98 seconds of confusion in the corpus, 71% fell in turns shorter than two seconds, and most of those were backchannels — yeah, right, mhm, for sure — spoken while somebody else held the floor. Another 19% fell within 500ms of a turn boundary, where one speaker's last syllable and the next speaker's first are being separated by a system that segments before it clusters. Long, uninterrupted, fully-voiced turns are where diarisation is most reliable, and an action item is almost always a long, uninterrupted, fully-voiced turn: the corpus average was 6.4 seconds.

Hand-labelling 214 action-item sentences across the eleven meetings and checking the attributed name against the truth gave 94.4% correct, twelve wrong. A sentence-level majority vote over the diarised words — assign the sentence to whichever speaker holds the most word-time within it, rather than to the speaker of its first word — recovered four of the twelve, because those failures were a backchannel from another participant clipped onto the front of a long turn. That change costs ten lines of code and moved the metric the product cares about by nearly two points while leaving DER exactly where it was.

from collections import defaultdict

def attribute_sentence(words) -> str:
    """Speaker with the most voiced time inside the sentence span."""
    totals = defaultdict(float)
    for w in words:
        totals[w["speaker"]] += w["end"] - w["start"]
    return max(totals, key=totals.get)

So the position to take into the launch meeting is neither "11% is fine" nor "hold until it is under 5%". Attribution on action items is 96.3% after the majority-vote change, the residual errors concentrate in a class you can describe, and the remaining question is what happens when the product is wrong. That is a design question rather than a modelling one. An action item rendered with the speaker's name as an editable chip, reassignable in one click from a list of the meeting's participants, converts a silent error into a two-second correction, and the participant list is known from the calendar invitation, which also bounds the speaker-count estimate that chapter five described as unstable.

Shipping it that way buys a measurement you cannot get any other way. Every correction a user makes is a labelled example arriving for free, and the correction rate on action items becomes a live production metric on real audio at full volume, rather than a quarterly number on eleven meetings. Set an alert on it, and treat a sustained rise as the same class of signal as a WER regression.

There is a case where this reasoning does not hold, and it is worth naming because it is the one that would justify the PM being overruled. If a wrong name causes harm that a later correction cannot undo — an attribution written into a compliance record, a disciplinary process, or a legal transcript — then the one-click correction is not a mitigation, because the damage happens at the moment of assertion. Products in that category should present speaker labels as suggestions in an interface that requires confirmation before the record is written, or should not present them at all until the recording infrastructure can supply channels.

What DER will not tell you#

It is a time-weighted measure, so a participant who speaks for 40 seconds in an hour contributes 40 seconds to the denominator and can be completely misattributed while the total barely moves. Jaccard error rate exists for this reason, computing an error per reference speaker and averaging over speakers so that everyone counts equally, and it is the better metric when the product cares about every participant rather than about total speech. Reporting both is cheap.

DER also says nothing about the words. A perfect diarisation attached to a transcript with 19% WER produces correctly attributed nonsense, and the two error sources compound in ways neither metric captures — the composite question, "what fraction of action items are both correctly worded and correctly attributed", is the only one the summary email actually depends on, and it has to be measured directly.

Twelve wrong attributions is also not where that meeting-notes product loses most of its credibility with users. The summary emails contained Priya rendered as Pria, Preeya and once as free ya, and the SOW in the action item was transcribed as ess oh doubleyou. Proper names and domain jargon carried a 34% error rate back in chapter three, and there is a mechanism designed specifically to attack that number.