Relaying a provider stream

The hop exists for an unglamorous reason: an API key cannot ship to a browser, so the provider call moves behind an endpoint you own. The component that results is a relay — a route that opens a streaming request upstream and writes the events onward to a client, holding two sockets at once for the duration of a generation. It is the first place in this topic where you are writing the server side of an event stream rather than reading somebody else's, and it inherits obligations the provider was quietly meeting on your behalf.

The one that surprises people is event framing. Server-Sent Events — the format introduced at the start of this topic, where an event is a run of field: value lines terminated by a blank line — says nothing about where one write ends and the next begins, and the property every client parser leans on turns out to be something the provider was supplying by accident.

The guarantee that disappeared when the hop appeared#

A provider writing an SSE body flushes once per event. It has just produced a text delta, it serialises the event, it writes it, and it flushes so the bytes leave immediately — that flush is the entire point of streaming. The consequence is that every write the provider makes ends exactly on an event boundary, and because TCP tends to deliver a small write as a single segment, every read the client performs tends to contain a whole number of complete events. That is an accident of implementation rather than anything the SSE specification promises, and it is why the sloppiest possible client parser works fine against a provider directly.

Each link in that chain is worth testing separately, because each one is a tendency rather than a rule and they fail independently. The provider flushes per event because a flush is what streaming means, so that link is solid. A small write leaves as a single segment because it is small — under the path's maximum transmission unit, the largest payload a single segment can carry — so that link holds while the events stay small and stops holding the moment something bigger goes past. A single segment produces a single read because it arrives on its own, which holds while the connection is quiet and stops holding when segments queue up behind each other on a busy path. None of the three is promised anywhere. Together they produce a guarantee that a client can be written against, and the client cannot tell the difference between a guarantee and three coincidences that keep happening.

Here is what gets written against them:

// The parser that survives a direct connection and dies behind a relay.
const text = new TextDecoder().decode(value);
for (const line of text.split("\n")) {
  if (!line.startsWith("data:")) continue;
  handle(JSON.parse(line.slice(5)));
}

There is no buffer here. Each read is parsed on its own and discarded. The first chapter of this topic explained why the buffered version is correct; this is the version teams actually ship, because it passes every test they write against the provider.

Now put a relay in the middle, written in the obvious six lines:

@app.post("/chat")
def chat(body: ChatRequest):
    upstream = requests.post(PROVIDER_URL, json=to_provider(body), stream=True)

    def pipe():
        for chunk in upstream.iter_content(chunk_size=8192):
            yield chunk

    return StreamingResponse(pipe(), media_type="text/event-stream")

Nothing is dropped, nothing is reordered, and the total bytes that reach the browser are identical to the bytes the provider sent. What iter_content(chunk_size=8192) removes is the property the client parser was leaning on. The relay reads whatever has arrived, slices it at 8,192 bytes regardless of content, and writes those slices. An Anthropic content_block_delta event — the event: line, the data: line with its JSON envelope, and the terminating blank line — runs about 130 bytes for four or five characters of text. A 600-token answer is roughly 400 of those events, about 52 KB, which the relay chops into seven pieces. A slice boundary lands inside an event unless it lands exactly on the two-byte terminator, so the odds of a clean split are about 1 in 65. Six or seven corrupt boundaries per long answer is the expected outcome, not a race.

Both of those figures come out of the same division, and it is worth doing them rather than taking them, because the second one is the one that decides whether this is a bug you will see. Fifty-two kilobytes of stream cut into eight-kilobyte slices gives six or seven pieces, and a set of pieces has one fewer join between them than there are pieces — so a long answer offers five or six places where a slice boundary falls in the middle of the stream. Each of those places is a byte offset chosen by a running byte count, and a running byte count has no idea what the bytes say. Relative to whichever event it lands in, the boundary is equally likely to sit at any of that event's hundred and thirty or so byte positions, and only two of them — the two bytes of the terminator — produce two halves that are each a whole event. Two in a hundred and thirty is about 1 in 65, which means each of those five or six joins is corrupt with odds of about sixty-four to one. Every long answer is damaged. The only randomness is where.

What makes it look intermittent is that the two halves of a split event fail differently.

The tail is the quieter half and the more dangerous one. A fragment that does not begin with data: is dropped by the guard without complaint, so a stored answer can be missing a word every 8 KB with no error anywhere in the system. Short answers — anything under 8 KB of SSE, which is about 60 tokens — fit in one read and never split at all, which is why FAQ-style questions in a smoke test always pass.

The guard is worth a second look, because it is a defence that appears to be working and is the thing doing the damage. startsWith("data:") was written to skip the lines that are supposed to be skipped: the event: line, the id: line, a comment. All of those are legitimately not payload, and dropping them is correct. A fragment that begins in the middle of a JSON envelope is also not a line beginning with data:, and the guard has no way to tell the two apart, because "does not start with data:" is true of both. So the code path written to ignore things that do not matter quietly ignores the thing that does, and it does so without logging, throwing or counting. The head of a split event announces itself with a stack trace; the tail is absorbed by a correct-looking line of defensive code.

The sixty-token figure is the same division read the other way, and it explains which questions your smoke test is allowed to ask. Eight kilobytes divided by about a hundred and thirty bytes an event is somewhere around sixty events, and an event near the start of an answer carries a few characters — so an answer short enough to fit inside one slice is an answer of roughly that many tokens. A one-paragraph reply about opening hours never reaches the first slice boundary and cannot split. Everything the relay was actually built for does.

Because the failure is a function of where reads land, it can be made deterministic instead of waiting for support tickets. The cheapest version is a unit test that feeds the parser a recorded provider response one byte at a time and asserts that the events come out identical to feeding it in one piece; anything that parses per read fails on byte two. A slightly more realistic harness yields random slices of 1 to 40 bytes, which reproduces both the head and tail failures within a few hundred iterations. Against a live service, curl --limit-rate 4k forces small reads on the wire and produces the same corruption in a browser without any test code at all. All three are worth more than a load test, because the bug is not a race and no amount of concurrency makes it more likely.

The reflex fix is to make the client buffer properly, and the client should buffer properly. It is not sufficient. Your API is a public interface; you do not control every consumer of it, and an interface that only works when the consumer is careful about something you never documented is an interface that will break.

Accumulate, split, re-emit#

The relay's job is the same buffering discipline the client does, moved one hop upstream: read whatever bytes arrive, hold them, and release only complete events.

Once the relay is parsing rather than piping, a second decision falls out for free: what it re-emits does not have to be what it received. Passing the provider's event shapes straight through makes your browser code a client of the provider's protocol, so a model switch becomes a frontend release, and every provider quirk — the [DONE] sentinel, the difference between a choices[0].delta and a content_block_delta — leaks into your UI. Emitting your own small vocabulary costs nothing at this point because you already hold the parsed event.

export async function POST(req: Request) {
  const upstream = await fetch(PROVIDER_URL, {
    method: "POST",
    headers: providerHeaders,
    body: JSON.stringify(providerBody),
  });

  if (!upstream.ok || !upstream.body) {
    return Response.json({ error: "upstream_unavailable" }, { status: 502 });
  }

  const encoder = new TextEncoder();
  const decoder = new TextDecoder("utf-8");
  const reader = upstream.body.getReader();
  let buffer = "";

  const stream = new ReadableStream({
    async pull(controller) {
      const { done, value } = await reader.read();
      if (done) {
        controller.close();
        return;
      }

      buffer += decoder.decode(value, { stream: true });

      let boundary: number;
      while ((boundary = buffer.indexOf("\n\n")) !== -1) {
        const raw = buffer.slice(0, boundary);
        buffer = buffer.slice(boundary + 2);

        const event = parseProviderEvent(raw); // returns null for pings/unknowns
        if (!event) continue;

        if (event.kind === "text") {
          controller.enqueue(
            encoder.encode(
              `event: delta\ndata: ${JSON.stringify({ text: event.text })}\n\n`,
            ),
          );
        } else if (event.kind === "stop") {
          controller.enqueue(`event: done\ndata: {}\n\n`);
        }
      }
    },
    cancel() {
      reader.cancel();
    },
  });

  return new Response(stream, {
    headers: {
      "content-type": "text/event-stream; charset=utf-8",
      "cache-control": "no-cache, no-transform",
      connection: "keep-alive",
    },
  });
}

Two things in that route are load-bearing beyond the buffering. The pull callback is only invoked when the runtime wants more data, which means the relay reads from the provider at the rate the browser reads from the relay. Backpressure — a slow consumer holding back a fast producer, so that the excess waits at the source instead of piling up in the middle — propagates through the whole chain without any explicit accounting. Writing the same logic as a while (true) loop that calls controller.enqueue as fast as the upstream can supply removes that property, and the queued bytes accumulate in your process's memory. With a phone on a weak connection reading at 20 KB/s and a provider streaming at 60 KB/s, a 90-second generation leaves you holding several megabytes per session, and the failure appears as memory pressure on the API node rather than as anything recognisable as a streaming bug.

The chain that gives you that property for free is worth following, because it is the difference between two loops that look equally reasonable. The runtime calls pull when the queue behind the client-facing response has room. Room appears in that queue only once bytes have been handed off to the response socket. The socket accepts bytes only as fast as the browser's receive window allows, which is the browser's own reading rate. And pull is the only thing in the route that reads the upstream body. Follow those four and the reading rate of the browser sets the reading rate of the provider, without a single line anywhere that mentions rates. The last step happens at the other end of the same mechanism: a relay that stops reading its upstream socket lets its own receive buffer fill, which closes its window to the provider, so the difference between the two speeds waits on the provider's side of the network rather than on your heap.

The while (true) alternative removes exactly the second step. enqueue accepts what it is given whether or not there is room, so nothing ever declines the next read, and the queue becomes the place the difference between the two rates lives. Nothing about that loop looks wrong in review — it reads every byte, writes every byte, and loses none of them.

The cancel callback is the other one. It fires when the client-facing stream is torn down, and calling reader.cancel() there is the first hop of the abort chain that the cancellation chapter later in this topic follows to the end. Omitting it is exactly the shape of leak that chapter is about.

The Python equivalent is an async generator rather than a pull source, and Starlette applies backpressure by awaiting each yielded chunk before asking for the next:

async def relay(body: ChatRequest) -> AsyncIterator[bytes]:
    async with httpx.AsyncClient(timeout=None) as client:
        async with client.stream("POST", PROVIDER_URL, json=to_provider(body)) as up:
            up.raise_for_status()
            buffer = ""
            async for piece in up.aiter_text():   # arbitrary boundaries, by design
                buffer += piece
                while "\n\n" in buffer:
                    raw, buffer = buffer.split("\n\n", 1)
                    event = parse_provider_event(raw)
                    if event is None:
                        continue
                    if event.kind == "text":
                        payload = json.dumps({"text": event.text})
                        yield f"event: delta\ndata: {payload}\n\n".encode()
                    elif event.kind == "stop":
                        yield b"event: done\ndata: {}\n\n"

Note that aiter_text is still handing over arbitrary boundaries; the fix is not to ask the HTTP library for a better chunking, because no chunking it offers corresponds to SSE events. Some libraries do ship an aiter_lines, which gets you closer but not all the way, since a single event is several lines and you still have to group them at the blank line. The provider SDKs go further and hand you typed events directly, and for a relay that only ever talks to one provider that is a reasonable place to stop hand-rolling.

One responsibility arrives with the hop that the provider was quietly handling before: keeping the connection warm during silence. Managed load balancers and platform proxies close connections that have carried no bytes for a while, commonly 60 seconds, and there are two ordinary situations where a token stream goes quiet for longer than feels safe. Prefill on a large prompt produces nothing at all while it runs, and a retrieval-augmented request — one whose prompt has had documents fetched from a search index pasted into it — with 40,000 input tokens can spend 4 to 9 seconds there before the first delta. A tool-calling turn is worse: the model emits a tool call, your server executes it, and if that tool is a slow database query or another model call, 30 to 60 seconds can pass with an open connection and no traffic. The remedy is the SSE comment line — write : ping\n\n on a timer of 15 seconds, reset whenever a real event goes out — and it costs six bytes per tick while making the difference between a stream that survives a slow tool and one that dies at exactly the idle timeout.

Two details of that timer are choices rather than conventions. The interval is several times shorter than the timeout it is defending against so that more than one tick fits inside the window, which means a tick delayed by a busy event loop or a long garbage collection pause costs nothing — the next one arrives well before the deadline. A timer set at the timeout itself would be a race with the thing it is racing. And resetting the timer whenever a real event goes out matters because the intermediary is counting bytes rather than pings: during a dense stream the deltas are already resetting its clock, so a heartbeat fired regardless would be pure noise at the exact moment the connection is least at risk.

The mirror of that is writing to a socket that has already gone. Once the client disappears, a write raises EPIPE at the OS level, surfacing as ERR_STREAM_WRITE_AFTER_END in Node or a ClientDisconnect in Starlette, and it arrives inside a loop being driven by an upstream that knows nothing about your client. Treating it as an exception to log is the wrong instinct; it is a signal that the work should stop.

The instinct is wrong for a reason worth having in words, because the same mistake is available at every hop for the rest of this topic. An exception normally means an operation could not be completed and should be retried or reported. This one means the operation is no longer worth completing, and retrying it is exactly the wrong response, since there is no reader on the other end and there will not be one on the next attempt either. Catching it, logging it at error level and continuing the loop is the shape most codebases reach for, and it produces a relay that keeps a provider generating, keeps a worker parked and keeps a pooled connection checked out on behalf of nobody, while filing a steady stream of alarms about it. Proxying somebody else's stream means inheriting their obligations in one direction and the client's in the other, and this is the one signal that travels the wrong way.

Three things to keep out of the relay#

Accumulating the answer is the first. A relay that collects every delta, joins the text, and writes one response at the end is a correct HTTP server and a complete defeat of the feature, and the surprising part is how easy it is to arrive at by accident — a well-meaning full_text += delta for logging, followed by a return of the joined string, reads perfectly in review. A later chapter in this topic deals with intermediaries that do the same thing to you without asking.

Semantics is the second. The temptation once you are parsing events is to make the relay smart: strip partial markdown, filter profanity, hold a delta back until a sentence is complete. Every one of those reintroduces latency at the exact point this chapter spent removing it, and holding until a sentence boundary in particular costs 200 to 600 ms per sentence, which is most of the benefit of streaming at all. Transformations that need the whole answer belong after the stream, applied to the stored message.

That figure is not a measurement of the filter; it is the length of a sentence, which is the thing being waited for. A sentence is a dozen or two tokens, and a model produces tokens at a rate that turns a dozen or two of them into a few hundred milliseconds. A relay holding back until the full stop therefore delays every token in the sentence by however much of the sentence is still to come, which is longest for the first word and shortest for the last. Compare that against the first-token improvement the whole topic is built on and the filter has spent most of it, on every sentence, for the life of the endpoint. The rule that follows generalises past markdown and profanity: any transformation whose input is a unit larger than a delta is a transformation that has to wait for that unit, and waiting is the one thing a relay exists not to do.

Durability is the third, and it is the one people assume they are getting. The relay holds no state that survives the request, so a deploy that recycles the node mid-generation loses the answer, exactly as it would with any transport. Whole-event framing buys correctness on the wire, not persistence.

What the relay does buy is a boundary you own. Upstream of it the provider's framing and event vocabulary are somebody else's decisions; downstream of it the protocol is yours, which means the browser can be given delta, done and error and never learn which provider produced them, or which of the transports in the previous chapter carried them there. That is worth having for its own sake, and it is what makes the remaining chapters possible: every mechanism from here on — a terminal sentinel, an in-band error, an abort carried upstream — is expressed in the vocabulary the relay defines.

Which leaves one question the framing does not answer. The status line went out with the headers, several hundred tokens ago, and it said 200. What is this relay supposed to write when the upstream stops sending?