The client audio leg wants UDP, and here is why

The same build behaved differently in two places. In the office it was well inside the budget from chapter 1, with a median turn of 940 milliseconds and no complaints. Demoed from a hotel over the guest wifi, it drifted: the first two turns were fine, the fourth had a noticeable lag, and by the tenth the agent was replying to things roughly two seconds after they were said, with no single stage in the trace looking slow. The gap was not in any of the stages. It was in the queue between them.

The hotel link was losing about 2% of packets in bursts, with a round-trip time hovering near 70 milliseconds. The client sent audio over a WebSocket, which is to say over TCP — the Transmission Control Protocol, the reliable byte-stream transport almost everything on the web is built on — and every one of those losses had a cost that the trace did not attribute to anything.

What one lost packet costs a live stream#

TCP guarantees that bytes arrive in order. When packet 407 is lost and packets 408 through 415 arrive safely, the receiving kernel holds all eight of them in its buffer and delivers nothing to the application until 407 has been retransmitted and received. That is head-of-line blocking, and for a file transfer it is exactly right — you want the file, and a 100-millisecond hiccup in the middle of it is invisible.

For a live stream it is the wrong trade in two ways. The stall duration is significant relative to the budget: a fast retransmit costs roughly one round trip, so about 70 milliseconds on that hotel link, and a loss with no subsequent packets to trigger duplicate acknowledgements falls back to the retransmission timeout, which on Linux has a floor of 200 milliseconds. During the stall, 20-millisecond audio frames keep being produced at the sender at a constant rate, so the queue behind the loss grows by one frame every 20 milliseconds.

The second way is worse, and it is the one that produced the drift. Once the retransmission arrives and the receiver delivers the backlog, the application has a burst of frames representing 200 milliseconds of audio that should have been played over the last 200 milliseconds. There is no way to play them faster without either discarding audio or time-compressing it, so the naive implementation simply plays them in order, starting now — and the stream is permanently 200 milliseconds further behind than it was before the loss. Every subsequent loss adds its own delay to the same accumulated total. That ratchet is what the demo felt like:

turn  1   uplink queue  0 ms   turn latency   940 ms
turn  3   uplink queue 180 ms  turn latency  1,120 ms   (2 losses, RTO once)
turn  6   uplink queue 410 ms  turn latency  1,350 ms
turn 10   uplink queue 1,080 ms turn latency 2,020 ms

Nothing in that table is a slow component. The model, the synthesizer and the endpointer all behaved identically on turn 10 and turn 1. What changed is that the caller's speech was reaching the server a second after they said it, and by the time anyone thought to look at the socket the call was over.

Losing the packet is the cheaper option#

WebRTC carries audio as RTP, the real-time transport protocol, which is a thin timestamped header wrapped around each audio frame and sent over UDP — the User Datagram Protocol, which hands each packet to the network once and never asks about it again, so there is no retransmission and no ordering guarantee. The two mechanisms built on top of that are the point of the exercise.

The jitter buffer holds a small amount of audio — typically 20 to 100 milliseconds, sized adaptively from the observed variance in arrival times — so that packets arriving slightly early or late still play at the right moment. It is a deliberate latency cost paid to absorb network variation, and it is bounded: when the network gets worse the buffer grows a little, and when it recovers the buffer shrinks back, often by playing out audio at a very slightly altered rate. The buffer does not ratchet, because it is allowed to give up on packets that have not arrived by the time their playout deadline passes.

Packet loss concealment handles what happens when it does give up. The decoder synthesizes a replacement for the missing 20 milliseconds from the spectral characteristics of the surrounding audio, and Opus — the speech and music codec WebRTC negotiates by default, a codec being the pair of algorithms that compress audio for the network and reconstruct it at the far end — additionally supports in-band forward error correction, where a low-bitrate copy of the previous frame rides along inside the current one, so an isolated loss can often be reconstructed rather than concealed. Isolated losses at 2% are essentially inaudible. Sustained loss above roughly 10% starts to sound like a bad mobile call — which is degradation a human immediately understands and works around by repeating themselves, rather than a conversation that silently slides out of sync.

The connection-setup row is where the last three acronyms live, and they are all consequences of the fact that both ends are usually behind a router doing network address translation, or NAT, which rewrites private addresses onto one public one and has no idea what to do with an unsolicited incoming packet. Interactive connectivity establishment, ICE, is the procedure by which the two ends work out a path anyway: STUN servers tell each side what its address looks like from the outside, and a TURN server stands in the middle and forwards media for the pairs that cannot reach each other directly.

The bandwidth row is a side benefit that becomes a primary one on cellular. Raw 24 kHz PCM16 — uncompressed audio, sixteen bits per sample and twenty-four thousand samples a second — is 384 kilobits per second in each direction, which a congested mobile uplink will simply not sustain, and the failure mode when it cannot is queueing — the same ratchet as above, from a different cause. Opus at 32 kilobits per second removes the problem rather than managing it.

Setting up the leg that matters#

For a client talking to a hosted realtime session, the negotiation is a single SDP exchange — session description protocol, the text document in which each side lists the codecs it speaks and the network addresses it can be reached on — and the audio never touches your application code:

const pc = new RTCPeerConnection();

// Remote audio: attach the track the model sends back.
pc.ontrack = (e) => { audioEl.srcObject = e.streams[0]; };

// Local audio: the AEC-enabled capture from the previous chapter.
const mic = await navigator.mediaDevices.getUserMedia({
  audio: { echoCancellation: true, noiseSuppression: true },
});
pc.addTrack(mic.getAudioTracks()[0]);

// Events (transcripts, tool calls, cancellations) ride a data channel.
const events = pc.createDataChannel("oai-events");
events.onmessage = (m) => handleEvent(JSON.parse(m.data));

const offer = await pc.createOffer();
await pc.setLocalDescription(offer);

const answer = await fetch("https://api.openai.com/v1/realtime/calls", {
  method: "POST",
  body: offer.sdp,
  headers: {
    Authorization: `Bearer ${ephemeralKey}`,   // minted server-side, short-lived
    "Content-Type": "application/sdp",
  },
});
await pc.setRemoteDescription({ type: "answer", sdp: await answer.text() });

Two structural details in that snippet are easy to miss. The audio goes over a media track and the control events go over a data channel, which means the cancellation message from the previous chapter and the audio it cancels travel on separate paths with separate reliability settings — a data channel is reliable and ordered by default, and that is what you want for events, since a dropped response.cancel is not recoverable by concealment. And the credential is an ephemeral key minted by your backend rather than your API key, because this code runs in a browser; the negotiation is client-to-provider, so your server's role is authentication and policy rather than media relay.

Catching the ratchet when you are stuck with a socket#

Plenty of systems cannot migrate this week, and the drift is detectable and partly treatable without changing transports. The detection is a timestamp on every audio frame at capture, compared against arrival time at the server, with the difference tracked per call rather than aggregated. You cannot read absolute one-way delay from that comparison because the clocks are not synchronized, but you do not need the absolute value — you need the slope. A call whose capture-to-arrival difference is 40 milliseconds higher on turn 10 than on turn 1 has 40 milliseconds of queue somewhere, and a call where it climbs monotonically is the hotel.

The treatment is to make the queue bounded on purpose. If the application-level send queue on the client exceeds some depth, drop the oldest frames rather than the newest, since old audio is worthless and new audio is what the caller is saying now. A caller who loses 200 milliseconds of their own speech to a deliberate drop is in the same position as a caller on a lossy UDP link, which is the position you wanted. What you must not do is let the socket's own buffer absorb it silently, and the value to watch in a browser is bufferedAmount on the WebSocket, which is the number of bytes queued and not yet handed to the network:

const FRAME_BYTES = 960;              // 20 ms of 24 kHz PCM16
const MAX_QUEUED = FRAME_BYTES * 10;  // 200 ms of slack, then start shedding

function sendFrame(ws: WebSocket, frame: ArrayBuffer) {
  if (ws.bufferedAmount > MAX_QUEUED) {
    metrics.incr("audio.frame_dropped");
    return;                           // shed the newest; the queue is the enemy
  }
  ws.send(frame);
}

That is a worse jitter buffer than the one WebRTC ships, and it is dramatically better than none. It converts an unbounded, invisible latency ratchet into a bounded loss you can count, which means the failure shows up in a metric during business hours instead of in a demo.

Where TCP is still the right answer#

The argument above is about the leg between a human's device and the first server, and it does not generalize. The link from your backend to a model provider runs between datacenters over well-provisioned paths where loss is on the order of 0.01% rather than 2%, and there a WebSocket is the better choice: setup is one upgrade request, there is no ICE negotiation to fail, no TURN infrastructure to run, and reliable ordered delivery of control events is what you actually want. Every serious voice stack is mixed for this reason, with WebRTC on the client leg and WebSockets between servers.

WebRTC is also not a latency improvement on a healthy network. On the office wifi where the demo worked, the two transports were within about 10 milliseconds of each other, and if you measure only from a good connection you will conclude the migration was pointless. The gain is entirely in the shape of the distribution under adverse conditions, which is another way of saying it shows up in the tail rather than the median.

The costs are real and are usually underestimated. ICE negotiation fails for a small fraction of users behind restrictive corporate firewalls, and those users need a TURN relay, which is a server you operate, that carries all their media, and whose bandwidth you pay for. Debugging a call that failed to connect means reading ICE candidate logs. Some environments — a server-to-server integration, an embedded device with no WebRTC stack, a telephony gateway — cannot use it at all, and for those a WebSocket with your own jitter buffer and a monitored queue depth is a legitimate engineering choice, provided you monitor the queue depth, because the ratchet is silent otherwise.

There is also a class of deployment where none of this is a decision. If the caller is on a telephone, the transport was chosen decades ago by people who were not thinking about your latency budget, and the leg from their handset to your media server has properties you can measure but not change.