Streaming

Read these first

• Chat API shape
• Prefill vs decode

What you will know

• Stream a response and render tokens as they arrive

  SSE framing, buffering, and proxy behavior can turn incremental model output into a single delayed blob or corrupt event sequence.

• Proxy a stream through your own API to a browser

  The relay must preserve event boundaries while coping with slow clients, intermediary timeouts, and disconnected sockets.

• Cancel an in-flight generation and confirm billing stops

  A client abort is not automatically an upstream abort, and providers may still charge tokens generated or buffered before cancellation takes effect.

• Handle a stream that errors halfway through

  Partial text may already be visible, so retries and UI state must avoid silently duplicating or presenting an incomplete answer as final.

• Explain why streaming changes nothing about total cost

  Delivery timing affects perceived latency, while input and output token accounting still determines spend.

Chapters

1. 1

   Server-Sent Events

   Server-Sent Events is the transport every major model provider streams over: one HTTP response whose body stays open and is written a small text message at a time, with a blank line as the only boundary that matters. This chapter covers the wire format, the buffered decoding loop that survives a read landing mid-event, the cost of re-rendering markdown four hundred times, and the cases where SSE is the wrong choice.

   Read chapter
2. 2

   Token accounting in a streamed response

   The same input and output tokens are billed whether a response is written once or in four hundred pieces, so streaming changes the arrival time of the first byte and nothing on the invoice. It does change where the numbers live: delta events carry text only, token counts appear in a separate usage-bearing event you may have to opt into, and a cumulative field summed rather than assigned double-counts on exactly the longest answers.

   Read chapter
3. 3

   Choosing between SSE and WebSockets

   Both ride one TCP connection over the same path, so a WebSocket frame does not reach the browser sooner than an SSE event; the axes that actually separate them are direction, framing, and what proxies, CDNs and load balancers do with each. This chapter works through the trade for a token-push endpoint, the operational bill a WebSocket carries, and the specific requirement — input arriving during generation — that justifies paying it.

   Read chapter
4. 4

   Relaying a provider stream

   Putting your own API between the provider and the browser looks like piping bytes, and a fixed-size read ends mid-event often enough to hand the client half a `data:` line. A relay has to accumulate upstream bytes, split on the blank-line boundary and forward complete events — while translating them into a vocabulary of its own, propagating backpressure, keeping an idle connection warm, and treating a broken client socket as a signal rather than an error to log.

   Read chapter
5. 5

   In-band errors and terminal sentinels

   A streaming response spends its status code on the first byte, so a failure at token 400 arrives as an event inside the body or as nothing at all. This chapter covers the four ways a stream can end, why a terminal sentinel is the only positive evidence that a message finished, what a relay emits and persists when the upstream breaks mid-body, and the availability metric that counts sentinels rather than status codes.

   Read chapter
6. 6

   Propagating a client abort upstream

   A Stop button that aborts the browser fetch, lets the route handler observe the disconnect and return, leaves the provider decoding to its natural stop and billing every token, because the outbound request is a separate connection with its own lifecycle. This chapter carries the abort hop by hop into the upstream call, in Web-standard and Node and Python forms, and proves it landed from the distribution of completion lengths rather than from the diff.

   Read chapter
7. 7

   Proxy buffering

   With buffering enabled an intermediary reads your response as fast as you can write it, holds it, and forwards it when the body completes — so the server log says the first token left at 300 ms and the browser shows nothing for nine seconds and then everything at once. This chapter covers what nginx is doing and why, the directives and headers that stop it, the two-run curl experiment that places the fault on one side of a line, and the serverless cut-off that looks identical and is not.

   Read chapter
8. 8

   Retrying a partially delivered answer

   A retry is safe while nothing has been shown and is an edit to a document somebody is reading after the first delta lands. This chapter draws that boundary, compares re-issuing the request against continuing from the delivered text on cost, latency and joinability, gives the turn state and the schema that keep two attempts at one answer straight, and settles when a second answer may be presented as though it were the first.

   Read chapter