Locating a buffering hop

Streaming can work for six months and then stop working without a line of your code changing, because the path a response travels is made of hops with their own defaults and their own release cycles. A typical arrangement puts an edge point of presence, a regional cache tier, a firewall, a platform router, an ingress controller and your pod between a browser and a token, and a team owns two of them. The earlier topic on streaming covered the single-proxy version of this, in which nginx sits in front of an origin you also run and the fix is a directive you can deploy. This chapter is the version where you own neither the hop nor its configuration and have to find it first.

The instrument for that is a probe: a route on the same service that emits chunks on a schedule you control, with the send time written inside each chunk, so that arrival can be compared against emission without a model's timing contaminating the measurement and without clock synchronisation between machines. Everything else here — the ladder of tests, the headers, the standing check — is built on having one.

Why anything would do this#

Buffering a response is the default in more places than teams expect, because for ordinary HTML and JSON it is the correct optimisation: hold the body, then write it to a slow client in one efficient burst, freeing the origin connection immediately. nginx ships with proxy_buffering on. A response body cannot be scanned by a web application firewall for injection patterns until enough of it exists to scan. A compressor works on blocks, and a gzip or brotli encoder at the edge will not emit anything until its window fills unless the application explicitly flushes at each chunk. Anything that rewrites a response — injecting analytics, minifying, transforming images — must have the response to rewrite it. And an intermediary that downgrades to HTTP/1.0 when talking to the origin, or that decides to set a Content-Length, has to buffer by definition, because it cannot know the length until it has the body.

Four of those six hops are capable of the behaviour, which is why a diagnosis that begins with a guess costs a deploy per guess and why "disable the CDN for that route" is a proposal to discard TLS termination, anycast routing and DDoS absorption in exchange for a fix that is usually one response header.

The signature in the takeaway follows from what buffering does to the schedule rather than to the bytes. An intermediary that collects a body forwards nothing until it has a reason to — a complete response, a full compression window, a finished inspection — and then forwards everything it holds as fast as the downstream connection will accept it. The client's first byte therefore arrives only once the origin's last byte has been produced, which makes time-to-first-byte at the browser approximately the whole generation time. The remaining chunks are already sitting in one buffer by then, so they arrive back to back with gaps set by bandwidth rather than by decoding. Neither half of that is observable from your process, because every latency your process records runs from a request arriving to bytes being handed to a socket, and the collecting begins after the last of those measurements has been taken. Leaving this diagnosis to server-side metrics fails in principle rather than merely in practice.

The compressor deserves one further step, because it is the hop that catches teams who have already disabled buffering everywhere else. A block compressor earns its ratio by referring backwards: a run of bytes is encoded as a pointer to an earlier occurrence of the same run, so the encoder cannot emit anything until it has accumulated enough input to know what repeats. Emitting after every write is possible, and every real implementation has a flush primitive for it, but it forfeits most of the compression — so the default in a general-purpose proxy is to fill the window first. That default is correct for the HTML and JSON it was written for, and on a stream it is indistinguishable from a whole-body buffer, which is how a text/* compression rule that happens to match text/event-stream produces the exact symptom this chapter opens with.

A probe with no model in it#

Diagnosing this with real traffic is slow and ambiguous, because every measurement is contaminated by the model's own timing. A 380-token answer at 45 tokens per second takes 8.4 seconds to decode, so an 8.4-second wait is consistent with both a buffering hop and a completely normal generation, and only the arrival pattern of the chunks separates them.

@app.get("/_probe/stream")
async def probe(count: int = 60, interval_ms: int = 1000, size: int = 40):
    async def gen():
        for i in range(count):
            sent = time.time()
            pad = "." * max(0, size - 60)
            yield f"data: {{\"i\":{i},\"sent\":{sent:.3f},\"pad\":\"{pad}\"}}\n\n"
            await asyncio.sleep(interval_ms / 1000)
    return StreamingResponse(gen(), media_type="text/event-stream")

The client side records arrival times and prints the delta between consecutive chunks.

curl -sN --http2 -H 'accept: text/event-stream' \
  'https://api.example.com/_probe/stream?count=20&size=40' \
| while read -r line; do
    [ -n "$line" ] && printf '%s  %s\n' "$(date +%s.%3N)" "$line"
  done

A healthy path steps once per second. A buffered path produces nothing for the full 20 seconds and then twenty lines with arrival timestamps 3 to 6 ms apart, each carrying a sent value one second apart from the last — which is the proof, because it shows the origin emitted them on schedule and something collected them.

Two choices inside that probe are doing more work than they appear to. Writing the send time into the chunk means no comparison ever crosses a clock boundary: the sent values are subtracted from one another and both came from the origin's clock, while the arrival timestamps are subtracted from one another and both came from the client's. Nothing subtracts one machine's clock from the other's, so the two hosts can disagree about the time by an hour without disturbing the result. The second choice is the absence of a model, which turns the emission schedule into a constant the test controls rather than a variable the provider sets. That is what upgrades the finding from "the answer arrived late" to "the origin emitted on schedule and something downstream held the output", and only the second of those names a hop.

The size parameter is what turns "something buffers" into a description of the buffer. Twenty 40-byte chunks is 800 bytes, which sits below every threshold, so a size-bounded buffer and a whole-response buffer look identical at that size. Re-running with size=4096 distinguishes them: if the chunks start arriving in pairs roughly every two seconds, the culprit is a fixed 8 KB buffer flushing when full, which is nginx proxy_buffer_size territory and is fixed differently. If nothing arrives until the response ends at any chunk size, the cause is compression or full-body inspection rather than a byte-count buffer.

Both sizes are chosen against the threshold they have to straddle. Twenty chunks of 40 bytes is 800 bytes in total, comfortably smaller than any buffer an intermediary is plausibly holding, so a size-bounded buffer never fills and behaves exactly as a whole-response buffer would. At size=4096 two chunks come to rather more than 8 KB, so a hop holding an 8 KB buffer fills it on roughly every second chunk and flushes — which is why arrival in pairs, about two seconds apart against a one-second emission interval, identifies a byte-count buffer and nothing else. A hop that emits nothing at any chunk size is not counting bytes at all. It is waiting on a condition that only the end of the response can satisfy, which is what both compression and full-body inspection are doing.

Working inward from the client#

With a probe that costs nothing to run, locating the hop is a matter of addressing each one directly and asking whether the behaviour survives. Each test is only interesting in relation to the one before it.

Two details make this sequence work in practice. Test with production-like request headers, because edge routing rules commonly key on accept-encoding, user-agent, cookies, or the accept header, and a curl with defaults can take a different path through the edge than a browser does. And test the same URL path, since edge configuration is usually path-matched: a probe mounted at /_probe/stream may miss a rule that applies to /v1/chat/*, so the probe is more honest when it lives under the same prefix as the endpoint it stands in for.

The headers that ask intermediaries to leave a body alone#

Fixing a buffering hop is a configuration change on that hop, and a streaming response can also state its requirements in ways that most intermediaries honour, which is worth doing on every streaming route regardless of what is currently in front of it.

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache, no-store, no-transform
Connection: keep-alive
X-Accel-Buffering: no

no-transform is the one that matters against compressors and rewriters: it is a directive in the caching specification that forbids intermediaries from altering the representation, and edge compressors generally respect it. X-Accel-Buffering: no originated in nginx as an application-controlled override of proxy_buffering and has been adopted by several CDNs and platform proxies as the conventional way for an origin to say "forward this as it arrives". Neither is universally honoured, which is why they are a supplement to the hop's own configuration and not a substitute for verifying.

Where the origin's own defaults are the problem, the fix is to disable compression for the route rather than to hope. A gzip middleware in your framework will buffer just as thoroughly as one at the edge unless it is configured to flush per write, and the bytes saved are irrelevant: SSE text compresses well, but a streaming answer is a few kilobytes and the bandwidth question is not close to the responsiveness question.

The probe as a standing check#

An investigation that ends with a corrected setting has fixed today's problem and left the next edge configuration change free to reintroduce it. The probe route is already the guard; what it needs is to be run without a human.

Point a synthetic monitor — a scripted request issued on a schedule from outside your network, rather than a measurement taken from real traffic — at /_probe/stream?count=10&interval_ms=1000 from each edge region you serve, every five minutes, with two assertions: time to first chunk under 1.5 seconds, and maximum inter-chunk gap under 2 seconds. Both thresholds are generous against a 1-second emission interval and neither can be satisfied by a buffered response, which will fail the first assertion by roughly the full ten seconds. Alert on two consecutive failures from the same region, because the region dimension is what turns "someone is complaining" into "the Sydney POP changed". The same probe belongs in the deployment pipeline, run against a preview environment that sits behind the same edge configuration as production, so that a change to a shield or WAF rule fails a check before it reaches customers.

Both thresholds come from the emission schedule rather than from a sense of what feels generous. Ten chunks at a one-second interval is a ten-second response, so a healthy path returns its first chunk almost immediately and never leaves more than a second between chunks, and the two assertions sit above those with room for ordinary jitter. A buffered path returns its first chunk only when the last one was produced, ten seconds in, so it misses the first assertion by roughly the whole ten — a margin no slow network or busy origin can manufacture. Putting the second assertion alongside it also catches the compensating case, where a hop forwards the first chunk promptly and then collects everything after it, which would satisfy a time-to-first-chunk check on its own.

The monitor should send the same headers a browser sends, including an accept-encoding that offers compression, since one configured with identity will pass cheerfully through exactly the failure described next.

Three things mistaken for a buffering hop#

Slow time-to-first-token from the model is the first. Both produce a wait before text appears, and they are trivially separable by the shape of what happens next: a slow model produces a late first token followed by a normal token cadence, while a buffering hop produces a late first token followed by everything else at once. Measuring only time-to-first-byte at the client conflates them, so the metric worth recording is the maximum inter-chunk gap alongside it.

Small-packet delay is the second, and it is the one people reach for first because it is the famous one. Nagle's algorithm interacting with delayed acknowledgements adds tens of milliseconds, and disabling it with TCP_NODELAY is sometimes worth doing, but a 40 ms effect does not produce an eight-second one. Similarly, HTTP/2 flow control can stall a stream, and that shows as chunks arriving in bursts as the window updates, not as a single blob at the end.

The third is the hope that a client can work around it. There is no request header a browser can send that makes an intermediary stop buffering; accept-encoding: identity is a diagnostic, not a fix, because you do not control what a customer's browser sends. Every real remedy is a configuration change at a hop, which is why the diagnosis has to name one.

What you now have, across this topic, is a set of independent controls that a production streaming endpoint needs at the same time, and they are independent in a way that matters operationally: each one fails without disturbing the others. The disconnect policy decides who owns a generation when the reader leaves. The durable chunk log decides whether the answer survives the process. The client's acknowledged sequence decides where replay begins. The bounded send queue decides what happens when the reader is slower than the decoder. The probe decides whether any of it reaches a human.

The way to use them is in that order, and incrementally. Instrumenting the gap between billed and delivered tokens costs an afternoon and tells you whether you have a disconnect problem worth a policy. Adding the chunk log is the largest change and is what everything after it depends on. Resumption is small once the log exists. Backpressure is a few lines and a metric that will sit at zero until the day it does not. The probe is a route and a synthetic check, and it is the only one of the five that will tell you about a failure you did not cause.