Streaming in production

Read these first

• Streaming

What you will know

• Handle a client disconnecting mid-generation without losing the output or the money

  The user closes the tab at token 300 of 900. Decide whether the generation dies or completes into storage, and make billing agree with that choice.

• Persist partial output so a reconnecting client can resume

  A durable sequence or cursor is needed to distinguish an acknowledged prefix from duplicated or missing tokens after retries and process failure.

• Identify a proxy or CDN that buffers the stream and defeats it

  Buffering proxies collect the whole response before forwarding, which silently defeats streaming end to end; verify behavior with timed chunks and production-like headers.

• Apply backpressure when the consumer is slower than the model

  Use a bounded buffer and an explicit policy for blocking, cancellation, or dropping work so a slow client cannot exhaust process memory.

• Explain what happens to a generation when nobody is listening

  Cancellation, provider retention, and billing semantics may diverge, so an abandoned request can continue consuming capacity unless ownership is made explicit.

Chapters

1. 1

   Abandoned generations

   A generation is a unit of work with its own lifetime: it starts when your server hands a provider a prompt and ends at a stop sequence or the max_tokens ceiling, whatever happens to the browser in between. This chapter separates it from the HTTP request that started it, covers the three ways a vanished client is actually detected and how far apart they are in speed, and prices what an abandoned generation holds — tokens, a worker, a pooled connection, and rate limit.

   Read chapter
2. 2

   Choosing a disconnect policy

   When a client vanishes mid-generation there are exactly two available responses: cancel the upstream call, or detach it and let it finish into storage under a generation id. This chapter states both precisely, prices the choice from the expected cost of not having the answer, gives the handler shape each one requires including the serverless case, and shows what each policy leaves on a cost dashboard when it is working and when it has silently stopped.

   Read chapter
3. 3

   The durable chunk log

   Treating a generation as an append-only log — every emitted chunk given a monotonic sequence number under a generation id, written durably before it is forwarded — turns the answer from a value one process is holding into a record that survives the socket and the pod. This chapter covers the minimum schema, why reversing the write and the flush loses tokens rather than duplicating them, what per-token writes cost and how coalescing fixes it, and why sequence numbers must come from one writer.

   Read chapter
4. 4

   Resuming from an acknowledged sequence

   Replaying everything stored re-renders text already on screen, and replaying from the server's own last write skips whatever died between the insert and the flush, so the boundary has to be the last sequence the client actually rendered. This chapter builds that cursor and the resume endpoint that serves backlog and live tail as one operation, prices the load it adds, and explains why exactly-once delivery is unavailable and idempotent application is the property worth engineering for.

   Read chapter
5. 5

   Backpressure on a slow reader

   A relay that forwards provider chunks without checking whether the write succeeded keeps pulling at decode speed while the client's send buffer is full, so undelivered chunks accumulate until the pod is killed. This chapter does the arithmetic that turns 15 events per second of difference into 250 MB an hour, honours the writable signal, sets out the three policies for a full queue with a bound and a metric, and gives four measurements that separate unbounded buffering from a genuine leak.

   Read chapter
6. 6

   Locating a buffering hop

   When six hops separate a browser from your pod and you own two of them, finding the one that collects the whole response before forwarding takes a probe rather than a guess. This chapter gives the signature that identifies buffering from the client side, a model-free probe route whose chunk schedule you control, the ladder of tests that addresses each hop in turn, the headers that ask intermediaries to leave a body alone, and the synthetic check that stops an edge configuration change from regressing it quietly.

   Read chapter