Speech to text

Read these first

• Chat API shape

What you will know

• Transcribe streaming audio with acceptable latency

  Measure time to first partial and final transcript under realistic network and audio conditions, because buffering can make an accurate recognizer feel unusable.

• Measure word error rate on your own domain audio

  A representative, manually corrected test set exposes accents, jargon, noisy channels, and proper-name failures that vendor benchmarks hide.

• Handle diarisation and timestamps

  Verify speaker attribution and segment boundaries against reference annotations, especially where overlapping speech or late revisions can misalign agent actions.

• Improve accuracy with a domain vocabulary

  Vocabulary hints can help rare terms but may increase false substitutions when the recognizer over-biases toward supplied words.

• Run a downstream agent with controlled transcription errors and measure task degradation

  Errors in negation, numbers, names, and turn boundaries can change tool calls or decisions far more than their aggregate WER suggests.

Chapters

1. 1

   What an 8% word error rate counts, and what it can exceed

   WER is substitutions plus deletions plus insertions, divided by the number of words in the reference. Insertions are not bounded by the reference length, so a recogniser that hallucinates through silence can score well above 100%. Work through the alignment on a real pair of strings, because the single aggregate number is the thing every later chapter has to argue with.

   Read chapter
2. 2

   A vendor's 4% describes the vendor's audio, not yours

   Published benchmarks are usually clean, wideband, read speech from cooperative speakers. Your call-centre audio is 8kHz telephony with codec artefacts, crosstalk, background noise and accents, which is most of the gap between an advertised 4% and the 19% you measure. Build a representative test set from your own channel, transcribe it by hand, and accept that a few hours of careful reference transcription is the price of knowing anything.

   Read chapter
3. 3

   The words a decision hangs on are not weighted like the rest

   WER treats 'the' and 'not' as equally costly, which is why an 8% recogniser can book the wrong appointment on one call in five. Dates, digits, names, negations and turn boundaries carry the task, and they are also the tokens most often mangled. Measure entity and slot error rate alongside WER, then inject controlled errors into a transcript and run the downstream agent to see what actually changes a tool call.

   Read chapter
4. 4

   Partials are hypotheses, finals are commitments

   A streaming recogniser emits a revised guess every 200ms and settles roughly 700ms after speech ends. Firing your LLM turn on the last partial buys back that gap, and pays for it when the partial gets rewritten by audio that had not arrived yet. Whether the trade is safe depends on whether the turn is cancellable and side-effect free, so measure time to first partial and time to final under realistic network and audio conditions.

   Read chapter
5. 5

   Speaker labels are inferred, except when they are recorded

   Diarisation clusters voice embeddings across the whole recording, so early labels get revised once more audio arrives, which makes them unreliable in a low-latency pipeline. Word timestamps have the same character: the decoder infers alignment rather than measuring it, and the drift grows on long segments. When a SIP provider records caller and agent on separate channels, transcribe each channel alone, because channel membership is a fact rather than a guess.

   Read chapter
6. 6

   Diarisation error rate sums three failures against reference speech time

   DER adds missed speech, false alarm and speaker confusion and divides by total reference speech time, where the reference is nothing more exotic than hand-marked speaker-labelled intervals: the diarisation equivalent of the transcripts you already produced by hand. The number moves substantially with the collar you allow around each boundary and with whether overlapped speech is scored at all, so 11% on crosstalk-heavy meeting audio read from a single mixed channel is an optimistic figure with no separate-channel escape hatch available. Then do for speakers what you did for words: measure the share of action-item sentences attributed correctly, since errors concentrate on short turns and boundaries, and ship the label as a one-click correction rather than a claim.

   Read chapter
7. 7

   Vocabulary hints buy rare terms and bill you on everything else

   Boosting 300 drug names drops their error rate from 41% to 12% by biasing the decoder toward them, and that bias does not switch off on audio where none of them were said. The measurement that decides whether to ship is the substitution rate on held-out audio containing no boosted term at all, where 'better' can quietly become 'Wellbutrin' in a sentence about the weather.

   Read chapter