Backpressure on a slow reader

In Node, res.write() returns false when the amount of data queued inside the stream exceeds its high water mark, which for an HTTP response socket defaults to 16 KB. That false is not an error and nothing is dropped; the data is appended to the stream's internal buffer and will be sent when the socket drains. The contract is that you stop writing until the drain event fires. Every streaming runtime has an equivalent — a promise that resolves when a writer is ready, a ReadableStream pull source that is only called when the consumer wants more — and in every case the signal is advisory, so ignoring it is legal and silent.

The signal has to be advisory, and that follows from how these runtimes work rather than from an oversight. A write that enforced the limit would have to block until the socket drained, and inside a single-threaded event loop a block is never a block on one connection: it stops every other connection the process is serving, along with its timers and its health check. The only enforcement available to a runtime that cannot afford to stop is a report. So write copies the data, returns a verdict, and leaves the decision to the caller. Ignoring that verdict is therefore not a bug in the framework and does not read as one in review — the loop that discards it is doing something the API explicitly permits.

Growth requires a producer faster than the consumer, and streaming from a model supplies one reliably. This chapter is about the arithmetic that follows, the three policies available when the queue is full, and how to tell the resulting memory curve apart from a leak, since both grow through the day and both reset on restart.

The arithmetic of a slow reader#

The provider decodes at about 45 tokens per second, delivered as one SSE event per token over a connection that is fast, close, and rarely congested. A phone on a weak mobile connection, or a laptop that has just been unlocked and is fetching a hundred other things, drains that response at perhaps 30 events per second. The 15 events per second of difference accumulate.

Each accumulated event is not 8 bytes of text. It is an SSE frame — id:, event:, data: with a JSON envelope and two newlines — which lands at about 120 bytes on the wire, plus the JavaScript string, plus the Buffer the stream allocates for it, plus the object in the write queue. Measuring retained size in a heap snapshot puts it at roughly 190 bytes per queued event. One stuck stream therefore accumulates about 2.85 KB per second.

The remaining number is how many streams are stuck at once. On a pod carrying 200 concurrent generations, with telemetry reporting that about 15% of clients disconnect before their answer finishes, those disconnected ones are the worst case: nobody is draining at all, so the full 45 events per second accumulate. Adding them to genuinely slow mobile readers gives roughly 26 streams accumulating at any moment, which is 74 KB per second, or 266 MB per hour.

That chain is four multiplications and one estimate, and the estimate is the part to hold loosely. A reader draining 30 events per second against a producer emitting 45 leaves 15 events a second with nowhere to go, and 15 events at roughly 190 bytes of retained size each is about 2.85 KB per second for a single stream. Twenty-six streams accumulating at that rate is 74 KB per second, and an hour being three thousand six hundred seconds makes that a little over 266 MB an hour. The 26 is the estimate: it blends genuinely slow readers with disconnected ones, and a disconnected reader is accumulating the full 45 events per second rather than 15, so pricing all 26 at the slow-reader rate makes 266 MB a floor rather than a central figure. That is the right direction for a number whose job is to decide whether a container limit is survivable.

// the version that OOMs
for await (const event of upstream) {
  res.write(sse(event));            // return value discarded
}

// the version that does not
for await (const event of upstream) {
  if (!res.write(sse(event))) {
    await once(res, "drain");       // reads from upstream stop here
  }
}

Awaiting drain inside the loop stops pulling from the provider's socket, which stops reading from that TCP connection, which lets the receive window close and eventually applies flow control upstream. Reads then proceed at the client's rate, and the memory held per stream is bounded by the high water mark rather than by the length of the answer.

The upstream half of that sentence is the part usually taken on faith, and it is two links long. Stopping the loop stops calling read on the provider's socket, so bytes accumulate in your kernel's receive buffer instead of being consumed. A full receive buffer is not a silent condition: the window your machine advertises to the sender shrinks toward zero, the provider's sending stack is obliged to respect it, and it stops putting bytes on the wire and starts holding them on its own side. The chain stops there for the purposes of this chapter, because the belief worth correcting lives at this level and not below it — and the belief is that the decoder is somewhere in the chain. It is not. It goes on producing into the provider's buffers at its own pace, which is why the last section of this chapter can say that backpressure protects the heap and leaves the invoice alone.

Bounding the buffer and choosing a policy#

Waiting for drain is one policy among three, and the right one depends on what the stream is carrying. The decision has to be explicit, because the default in every framework is the unbounded one.

Coalescing is the option that looks reckless and is not, provided the chunk log from earlier in this topic exists. Instead of queueing 40 undelivered per-token deltas, merge them into one event carrying the concatenated text and the sequence number of the last delta in the batch. The reader gets a bigger jump of text, the sequence numbering stays monotonic so the resume cursor still works, and the fine-grained truth remains in the log for anyone who resumes. What you must not do is discard events without merging, because a gap in the sequence turns "replay strictly after my cursor" into a request for chunks the client never received.

Whichever policy you pick, the queue needs a hard bound and a metric. A per-stream limit of 64 KB or 128 pending events, exported as a histogram of queue depth, converts the whole class of failure from an invisible slope into a dashboard line that moves before anything breaks.

const q = new BoundedQueue({ maxBytes: 64 * 1024, onFull: cfg.backpressurePolicy });
metrics.observe("stream_send_queue_bytes", q.bytes, { pod, tenant });

Telling buffering apart from a leak#

A memory leak and unbounded buffering produce the same graph: monotonic growth through the day, a hard reset at restart, no errors. The distinction matters because the remedies share nothing, and there are four measurements that separate them without a profiler.

Draining traffic is the fastest. Move a pod out of rotation and let its in-flight generations finish, without killing it. Buffered memory is released as streams complete or their sockets close, so resident memory falls back towards baseline within a couple of minutes. A leak holds; the graph stays flat at whatever it had reached.

Correlation says the same thing continuously. Plot heap growth against concurrent streams and against the share of traffic from slow networks, rather than against uptime. Buffering tracks the stream count and the slow-client share and disappears overnight when traffic does. A leak tracks requests served, cumulatively, and does not care whether they were streams.

Instrumenting the queue makes it arithmetic rather than inference. Export the sum of queued bytes across all active streams. If that sum accounts for the growth — 74 KB per second, matching a 250 MB per hour slope — the question is answered. If the sum sits flat at a few megabytes while resident memory climbs, you have a leak, and now you know it without a heap snapshot.

A snapshot settles anything left over. Retained size dominated by strings and Buffer instances reachable from response objects is buffering; retained size dominated by entries in a map that nothing ever deletes, or by closures held by an event emitter with a growing listener count, is a leak. Both can be present at once, which is the case where the first three measurements are most useful, because they tell you the proportions.

Where flow control is the wrong control#

Honouring writability does not reduce your bill. Closing the receive window against the provider does not stop the decoder; it stops bytes moving, while the provider's side buffers the response and the meter keeps running. Backpressure protects the heap, and cost is protected by the disconnect policy — the two are independent, and a system that applies backpressure diligently while never cancelling abandoned generations has a stable memory graph and the same invoice it always had.

It is also not rate limiting, which is the confusion that sends teams to the wrong control. Rate limiting is admission control at the front door: it decides how many generations start. Backpressure is flow control inside a generation that has already started. A concurrency cap of 150 streams per pod would have made the kill above take longer to arrive and would not have removed the unbounded growth, because the bound is on how many streams accumulate, not on how much each one accumulates.

And the writable signal is a coarse instrument. It tells you the kernel's send buffer and the TCP window have room, which means the network accepted the bytes — not that the browser parsed them and certainly not that anything was rendered. The gap between "the socket took it" and "the reader has it" is exactly the gap the acknowledged sequence from the previous chapter exists to close, and SSE offers nothing between the two. This is the structural complaint in AI Token Streaming: From SSE to Durable Sessions: server-sent events give you a one-way pipe with no application-level flow control, so everything you know about the reader's progress you have to build on top.

The whole mechanism is therefore one conditional and one bound. The conditional is the return value you were already being given and were throwing away, and honouring it converts a memory curve into a rate that matches the slowest reader. The bound is what you choose when honouring it is not enough — block, cancel, or coalesce — and the reason to choose deliberately is that the queue depth is the one quantity in a streaming system that grows without any event to attribute it to.

With writability honoured and the queue bounded, the last stall is the one your process cannot observe at all: an intermediary that accepts every byte you write, immediately and eagerly, and forwards none of them until the response is complete. Every metric on your side looks perfect while nobody sees a token for eight seconds.