Vocabulary hints buy rare terms and bill you on everything else

A clinical documentation product transcribed dictated visit notes and got drug names wrong 41% of the time, which made the feature close to useless — a note that says the patient is on metformin when she is on metronidazole is worse than no note. The team uploaded the 300 most-prescribed generic and brand names from the formulary — the approved list of medications the organisation actually stocks and prescribes — as vocabulary hints, re-ran their evaluation set, and watched the error rate on those terms fall to 12%. The ticket was closed and the change went to production on a Thursday.

The following week a physician's note read the weather has been Wellbutrin it got cold and her knee flared up. What she had dictated was the weather has been well, but then it got cold. Nobody had measured what the boost did to audio containing no drug name at all, and the sentence about the weather is the shape of the bill.

The bias is applied during the search, not after it#

Vocabulary hints work by changing the score of hypotheses while the decoder is still choosing between them. The classical implementation is shallow fusion with a contextual language model: the phrase list is compiled into a trie — a prefix tree, in which the words sharing an opening sequence of sounds share a branch, so the decoder can ask in one step whether the token it is considering continues any phrase on the list — or into the weighted finite-state transducer that generalises it. As the beam search from chapter four extends its candidate transcripts, any path whose next token continues a phrase in the trie receives an additive bonus to its log probability. Paths that begin a phrase and then abandon it have the bonus subtracted again, so a partial match is not rewarded. End-to-end models use attention-based variants of the same idea, where an encoded representation of the phrase list is attended over during decoding, but the character of the mechanism is unchanged.

What matters for the failure above is that the bonus is unconditional. The decoder is not asking whether a drug name is plausible in this sentence; it is adding a constant to every path that spells one. When the acoustics genuinely favour well but then, and the boost adds enough to Wellbutrin to overcome that margin, the boosted word wins. The bias has no notion of topic, no off-switch on non-clinical audio, and no awareness that the surrounding words are about the weather. Every request carrying the phrase list gets the bias, uniformly, across the entire audio.

The API surfaces differ in vocabulary more than in substance:

// Google Cloud Speech-to-Text: phrase set with per-phrase boost.
{
  "config": {
    "languageCode": "en-US",
    "adaptation": {
      "phraseSets": [{
        "phrases": [
          {"value": "metronidazole", "boost": 15},
          {"value": "Wellbutrin",    "boost": 8},
          {"value": "sotalol",       "boost": 15}
        ]
      }]
    }
  }
}

# Deepgram: keyterms on the request; the boost is per term, sign included.
options = {
    "model": "nova-3",
    "keyterm": ["metronidazole", "sotalol:2", "Wellbutrin:1"],
}

# Whisper-family models: biasing via decoder prompt, which is leakier still.
result = model.transcribe(
    audio,
    initial_prompt="Medications discussed: metronidazole, sotalol, Wellbutrin.",
)
# The prompt conditions generation, so the model can also emit prompt text
# verbatim into the transcript, and long prompts raise hallucination rates.

Prompt-based biasing deserves a specific warning, because it looks like the cheapest option and behaves the worst under stress. The prompt occupies the same context the model generates from, so on silence or noise the model can reproduce the prompt's own words as transcript — the hallucination behaviour from chapter one, now with your formulary as its source material.

The sweep that tells you where to stop#

Boost strength is a single knob with two opposing effects, and the only way to place it is to measure both curves on the same audio. Score two sets: a positive set of utterances that contain boosted terms, on which you measure the error rate on those terms, and a negative set of utterances containing none of them, on which you measure how often a boosted term appears in the transcript anyway.

The 12% the team shipped is the boost-20 row, and the same row costs 4.7 false drug names per 1,000 words on audio where no drug was mentioned. A typical fifteen-minute dictation runs about 1,800 words, so that setting inserts a spurious medication into roughly eight notes in every ten. The boost-10 row gives up six points of term accuracy and cuts the false rate by more than three times, and it is the only row where overall WER improves rather than degrades. Overall WER is a useful tripwire here precisely because it is the flat, unweighted metric from chapter one: when a boost setting starts hurting it, the bias has spread well beyond the terms it was meant to help.

Building the negative set takes an hour. Sample utterances from the same channel, check against their references that no boosted term and no inflected form of one occurs, and then score against those references while counting a specific error class — substitutions whose hypothesis word belongs to the boost list. The alignment operations stored in chapter one already contain everything needed.

BOOSTED = {t.lower() for t in phrase_set}

def false_boost_rate(scored_utterances) -> float:
    """Boosted terms hallucinated into audio that contained none."""
    hits = ref_words = 0
    for utt in scored_utterances:            # negative set only
        for ref_w, hyp_w, op in utt.ops:
            if op in ("substitution", "insertion") and hyp_w.lower() in BOOSTED:
                hits += 1
        ref_words += utt.ref_len
    return 1000 * hits / ref_words

# boost=20 -> 4.7 per 1,000 words
# boost=10 -> 1.4 per 1,000 words

Log the offending pairs, not just the rate, because the list is short and immediately actionable. At the clinical product it was dominated by five terms whose surface forms collide with ordinary English: Wellbutrin against well but then, Lyrica against lyric, Aleve against a leave, Celexa against sell exa fragments, and Bactrim against back trim. Removing those five from the phrase list and handling them through post-processing dropped the false rate at boost 20 from 4.7 to 1.1 per 1,000 words while costing almost nothing on the positive set, because those five were not the terms that were failing.

Fewer terms, scoped tighter, beats more terms#

Adding more vocabulary is the reflex when the positive-set number is still unsatisfying, and it works against you in both directions. A 5,000-term full formulary at boost 10 measured 21.3% term error rate and 5.9 false boosts per 1,000 words — worse on both axes than the curated 300-term list at the same boost. Longer lists dilute, because more of the search space is reachable through some boosted path, so the bias is spread over many candidates and the decoder is nudged toward the phrase list in general rather than toward the specific term that was said. Implementations also cap list sizes and degrade quietly past the cap.

The intervention with the best return is not tuning the boost at all, it is shrinking the list per request. A visit note is dictated about one patient, whose active medication list is ten to fifteen items and is sitting in the electronic health record — the EHR, the clinical system of record the product is already integrated with and has open on screen.

Twelve terms at boost 20 gave a term error rate of 9.4% and a false boost rate of 0.9 per 1,000 words, which beats every row in the sweep table on both axes at once. The reason is not subtle: the bias is now conditioned on context the decoder never had, supplied by a system that knows which drugs this patient actually takes.

The alternatives it is competing with#

Post-ASR resolution is the one most often skipped and it composes well with light boosting. Rather than forcing the decoder to output sotalol, let it output whatever it heard, then match candidate spans against the formulary by phonetic distance — Double Metaphone or an edit distance over phoneme strings — and resolve using the word confidences and the n-best list the API already returns, that list being the handful of alternative transcripts the decoder's search ranked just below the one it emitted. The advantage is that the correction happens where context is available and where a low-confidence match can be surfaced to the clinician for confirmation, and the transcript itself is never corrupted by a guess. The limit is that it only rescues terms the decoder came close to; a word rendered as something phonetically remote is unreachable.

Boosting also cannot help where the information is gone. Two drug names that are genuinely confusable in narrowband audio — the pairs that pharmacy safety literature tracks precisely because humans confuse them too — will remain confusable after biasing, and boosting both merely moves which one wins. Nothing in this chapter is a substitute for a validation step against the record: a medication mention that does not resolve to something plausible for this patient should raise a question rather than be written down.

Where this leaves the pipeline#

The order in which to do this work follows the order of the chapters, and it is not the order most teams choose. Build the representative test set first, because every other decision here is scored against it and a vendor's published figure describes audio that is not yours. Then define the entity and slot metrics that correspond to what your product actually does with the words, since an aggregate WER that looks acceptable can sit on top of a failure rate that is three times higher on the tokens carrying the task. With both in place, model choice, boost strength, streaming configuration and diarisation policy stop being matters of opinion.

The architectural decisions are worth making explicitly and early, because they are expensive to revisit. Which surfaces need streaming and which can be batch, given that batch is cheaper and one to three points more accurate on the same audio. Whether the recording infrastructure can be made to hand you one channel per speaker, which removes a whole category of inference. Where the audio tap sits relative to transcoding and gain control. What context your application can supply to each transcription request, and how narrowly it can be scoped. Those four choices bound what every metric in this topic can ever report, and no amount of measurement after the fact moves them.

Finally, treat every number in this topic as attached to a configuration rather than to a system. A WER without its normaliser, a DER without its collar and overlap policy, a latency without its network profile, and a boost result without its negative set are all claims that cannot be checked. Writing the configuration next to the number, in the dashboard and in the document, is what makes the next person's comparison meaningful — including when the next person is you, six months later, deciding whether a vendor's new model is actually better on the audio you have.