Speech-to-speech trades away the text checkpoint

The dental group's requirements are three sentences long. The agent calls patients to confirm appointments, it reschedules against a live calendar, and it must never state a price, because quoting a figure that turns out to be wrong after insurance has been applied is a complaint the practice has had before and a regulatory problem in two of the states it operates in. The prototypes are built. The speech-to-speech version — a single model that takes the caller's audio in and produces the reply as audio directly, with nothing written down in between — answers in 320 milliseconds median and sounds like a person, including the small breaths and the rising intonation on a question. The chained version — speech-to-text, then the language model, then text-to-speech, written STT, LLM and TTS in the table further down — answers in 900 milliseconds and sounds like an automated system, which is what patients will describe it as.

Choosing on those two numbers alone gives you the first one, and the argument for it is genuinely strong: 320 against 900 milliseconds is the difference between the window humans expect and the window they notice, and no amount of prompt work makes the chained version sound unrehearsed. The reason the decision is not that simple is the third sentence of the requirements.

Where a check can attach, and where it cannot#

In a chained pipeline the model produces text, and that text exists as a string in your process before any audio is made. Every mechanism that inspects, blocks, redacts or scores the agent's output — an output guardrail, meaning a check that runs on what the system is about to say and can rewrite or suppress it before anyone hears it — attaches at that moment, and it attaches deterministically because the string is right there.

PRICE = re.compile(r"(\$\s?\d|\d+\s?(dollars|usd)|costs?\s+\w+\s+dollars)", re.I)

async def speak(text: str, session) -> None:
    if PRICE.search(text) or await classifier.flags(text, "pricing"):
        metrics.incr("guardrail.blocked", tags={"rule": "pricing"})
        text = ("I'm not able to quote prices — the front desk can confirm "
                "the cost with your insurance. Shall I note that you asked?")
    session.history.append({"role": "assistant", "text": text})
    audit.write(call_id=session.id, turn=session.turn, text=text, checked=True)
    await tts.stream(text, session.player)          # audio is made after the check

Four separate concerns are satisfied by the position of those lines. The guardrail blocks the utterance before the patient hears any of it. The audit log records exactly what was said, as text, which is what a compliance review will ask for. Offline evaluation — the evals in the table below, meaning a fixed set of inputs replayed through the system with every output scored against what it should have been — runs against the same strings, so a regression suite of 300 adversarial prompts can be scored in seconds without synthesizing anything. And redaction of anything sensitive happens before the text reaches your logging pipeline, rather than being attempted on an audio file afterwards.

A speech-to-speech model produces audio tokens from audio tokens, those tokens being short discrete units of sound that the model reads and writes directly, in the way a text model reads and writes word pieces. There is no intermediate string that the audio is rendered from, so there is no moment at which the above function could run. You do get a transcript — providers emit one alongside the output — and it is a description of what the model is saying, produced concurrently with the audio rather than before it. By the time a rule matches the word "dollars" in that transcript stream, the corresponding audio has been sent, and quite possibly played.

That does not leave you with nothing, and it is worth being precise about what it leaves you with. You can watch the streaming transcript and cancel mid-utterance using exactly the machinery from chapter 4, which means the patient hears the beginning of a violation and then the agent stopping. Detection latency here is real: transcript deltas trail the audio by 100 to 300 milliseconds, cancellation takes another 50 to 150 to reach the client and flush, so the exposure window is something like a third to half a second of speech. "That'll be about two hun—" is a sentence the patient heard, and the incident report will say so.

The ASR-errors row cuts both ways and is easy to misread. A chained pipeline shows you that the recognizer heard "Tuesday the fifteenth" when the patient said "Tuesday the fiftieth", and you can add a confirmation step for low-confidence dates. A speech-to-speech model never makes that particular mistake in the first place, because it hears the audio and the ambiguity is preserved rather than collapsed into a wrong string — which is a genuine quality advantage, and simultaneously means there is no confidence score to gate on when it does misunderstand.

Deciding by what the product must prove#

The way to break the tie is to ask what the system has to be able to demonstrate to someone who is not in the room. For this dental group, that is: no patient was quoted a price, and here is the record. A detect-and-cancel design can be made to violate that requirement rarely, and rarely is a different category from never. Over 4,000 calls a month, a mechanism that leaks a third of a second of speech on 0.3% of the turns where pricing comes up is twelve incidents a month, each one a recording that exists.

There is a second design that gets most of the latency and satisfies the requirement, and it is worth reaching for before the tie is broken by architecture. Enforce the constraint by exclusion: the realtime session gets no pricing data in its context, no tool that returns a price, and a system instruction to route cost questions to the front desk. A model cannot state a figure it has never been given, and the failure mode that remains — inventing a number — is a hallucination, a fluent statement with nothing behind it, and the transcript guard is well suited to catching that because the model has no reason to produce one. That reduces the problem from "prevent a capable system from saying something it knows" to "catch a rare fabrication", which is a much easier thing to prove with an eval set.

Whether it is enough is an empirical question and the answer belongs in a test rather than an argument. Run both prototypes against the same 60 recorded calls with the same tool implementations, plus a set of 200 adversarial turns where a simulated patient asks about cost eight different ways, and fill in one table.

Numbers like these decide the question in a way that a preference for latency does not. Two violations in 200 adversarial turns is a rate the practice's counsel will not accept, and the follow-up work is knowable: tighten the instruction, add the transcript guard, re-run, and see whether it reaches zero and stays there across a larger set. The 1.6% wrong-date rate is the more interesting number, because it is a tool-control problem rather than a guardrail one, and it is the kind of thing that gets missed when a comparison is run on latency alone.

Design the fallback while you still have a choice#

Whichever way that goes, the realtime session becomes a single dependency carrying an entire product, and it is a young dependency with a young operational history. Provider incidents happen, regional capacity runs out, and a model version gets deprecated with a migration window measured in weeks. A voice agent whose only path is one vendor's session endpoint is a product that is entirely down during those events, and "entirely down" for an outbound reminder service means a day of appointments nobody confirmed.

The staged pipeline is the natural fallback because it is assembled from parts with independent failure domains, and it is only a real fallback if three things are true. The tool implementations are shared, so the calendar logic is one code path rather than two that drift. The conversation state — history, truncation, turn ids — has a representation both paths can read, which is straightforward if you kept your own history rather than relying entirely on the provider's session state. And the fallback is exercised on a schedule, with some small percentage of live traffic routed through it every day, because an untested failover is an assumption.

The economics support the same structure. At 71 cents a call against 14, a hybrid where routine confirmations run chained and only the conversationally difficult calls use the realtime session is a defensible position on cost alone, and it has the pleasant property that the fallback is warm because it is carrying traffic.

Building the thing#

The material across these chapters composes into a build order, and it is roughly the order they were written in. Write the budget with a number against every stage and instrument it before anything else, because you cannot tune what you cannot attribute. Get turn detection conditioned on content rather than on a single silence threshold, and evaluate it against annotated audio from the population you actually serve. Verify the echo path on the worst device your users have, then make barge-in a full cancellation with history truncation, because that is the piece that quietly corrupts conversations. Choose the transport for the leg you control and measure the floor on the one you do not. Then keep the per-turn traces, look at the cohort that complains rather than the median, and cover the delays that are genuinely irreducible without promising anything you cannot deliver.

The comparison in this chapter is the one decision that will be different in a year, because the audio models are improving quickly and the tooling around them — evaluation, guardrails, observability for audio-native systems — is being built now. Pipecat, the open-source framework for assembling voice agents out of these stages, is worth running end to end for a working reference of the pipeline, and the Realtime API guide is the spec to read closely before committing to a session-based design. What is unlikely to change is that a voice turn is a chain of deadlines, and that every architecture is judged on the turns where one of them was missed.