Proxy buffering

Proxy buffering is the behaviour named by nginx's proxy_buffering directive, which is on by default, and by its equivalents in most other intermediaries. When a response comes back from an upstream, nginx reads it as fast as the upstream can produce it, holds it in memory — proxy_buffers defaults to 8 buffers of 4 or 8 KB, so 32 or 64 KB — spilling to a temporary file on disk beyond that, and forwards to the client on its own schedule. For a static asset or a JSON API this is a real optimisation: it frees the upstream worker immediately rather than pacing it to a slow client, which is the backpressure problem from the relay chapter solved by absorbing it.

For an event stream it defeats the entire point. A 700-token answer is about 3 KB of text and perhaps 12 KB of SSE framing, which fits comfortably inside 32 KB of buffers, so nothing is forwarded until the upstream response completes. The symptom is unmistakable once you know it: your log records the first delta written at 312 ms, and the browser's performance timeline records the first byte of the response body at 9,140 ms and the last at 9,190 ms, with the entire answer painted in one frame. The application is fine. The bytes were written early into a socket that was read promptly by an intermediary which then declined to pass them on. This chapter is about which intermediary, how to prove it, and what to send so that most of them behave.

What nginx is doing with your body#

Buffering is not a misfeature, and knowing what it buys matters here because the remedy is to switch off something that is doing a real job. The origin worker is the expensive resource in most architectures: it holds a thread or an event-loop slot, a database connection, whatever the request needed to be answered. A client on a slow link takes a long time to accept a response, and an origin pacing itself to that client holds all of that for the client's transfer time rather than for its own processing time. A proxy that reads the body as fast as the origin can write it, and pays it out at whatever rate the client manages, moves the waiting from the origin to itself.

What is being moved there is precisely the backpressure the relay chapter propagated deliberately. The proxy takes the difference between a fast producer and a slow consumer into its own memory so that nothing upstream has to notice the difference exists. For a JSON document that is a good trade, because the only byte anybody wants is the last one and nothing is lost by the earlier ones waiting. For a stream it is the wrong trade by construction, because the earlier bytes are the entire product.

Which of the two shapes you get is decided by one comparison, and it is worth doing because it tells you what a long answer will do before you test one. A 700-token answer is about 3 KB of text, and the SSE framing around it — an event: line, a data: line carrying a JSON envelope, a blank line — costs several times the text it wraps, so the body on the wire is perhaps 12 KB. That total sits inside 32 KB of buffers with room to spare, so nothing is ever forced out early and the client sees one delivery at the end. An answer several times longer overflows the buffer set part-way through, the proxy flushes what it is holding and starts filling again, and the client sees two or three lumps. Same mechanism, different answer length. The lumps are worth recognising, because a slow model does not deliver in lumps.

No timeout setting affects any of this. With an infinite timeout the answer still arrives all at once, and that distinction is the load-bearing part of the diagnosis, because "raise the timeout" is the first suggestion in every thread on the subject and it is an answer to a different problem.

The two settings are watching two different clocks, which is the reason one cannot substitute for the other. A timeout answers "how long am I willing to wait for bytes that have not arrived", and it is armed against an upstream that has gone quiet. Buffering answers "when do I release bytes I already have". The proxy in this failure is not waiting for anything: it received the first delta at 312 ms, exactly as your own log says it did, and it is holding it. Raising a limit on waiting changes nothing about a component that is not waiting. Set the timeout to infinity and the answer still arrives in one piece at the end; set it too low and you have converted a latency bug into a truncated response, which is strictly worse.

The configuration for a streaming location:

location /api/chat {
    proxy_pass http://app_upstream;

    proxy_buffering off;
    proxy_cache off;
    gzip off;

    proxy_http_version 1.1;
    proxy_set_header Connection "";

    proxy_read_timeout 300s;
    proxy_send_timeout 300s;
}

proxy_http_version 1.1 matters more than it looks. nginx talks HTTP/1.0 to upstreams by default, which has no chunked transfer encoding, so the upstream is pushed toward buffering the body itself in order to compute a content-length. The chain is worth walking, because this is the one line in the block whose absence produces a bug rather than a sub-optimal setup. A response body has to be delimited somehow. HTTP/1.1 offers two ways — a declared length, or chunked encoding, where each piece announces its own size — and HTTP/1.0 offers only the first, plus the fallback of closing the connection. An upstream told it is speaking 1.0 therefore has to know the length before it writes the headers, and knowing the length before writing means holding the whole body. The buffering you removed at the proxy reappears at the origin, one hop earlier, with the proxy_buffering off directive still in place and still innocent.

Setting Connection to an empty string clears the close header that would otherwise be forwarded. gzip off is there because compression modules accumulate input to compress it, which reintroduces exactly the delay you removed — this is also true of Express's compression() middleware, which will hold an SSE stream indefinitely unless you call res.flush() after every event or exclude the route.

When you do not own the nginx configuration, the application can ask per response, by adding one header to the set the relay already writes:

X-Accel-Buffering: no
Cache-Control: no-cache, no-transform
Content-Type: text/event-stream; charset=utf-8

nginx reads X-Accel-Buffering from the upstream response and disables buffering for that response alone. Several CDNs and platform proxies honour it too, and no-transform asks intermediaries not to recompress. Setting these costs nothing and removes the most common instance of the problem without a configuration change anyone has to review.

nginx is the one people meet first, and it is not the only hop with an opinion about response bodies. A CDN in front of the origin may buffer unless it recognises the content type, which is one practical reason to send text/event-stream rather than text/plain even though the bytes are identical. Service meshes — the layer of sidecar proxies that carries traffic between internal services — based on Envoy stream by default but will buffer any route where a filter needs the whole body, and a WAF rule that inspects response content is exactly such a filter. Apache's mod_proxy needs flushpackets=on for the same reason nginx needs proxy_buffering off. Managed application load balancers generally pass bodies straight through, while older API gateway products designed around request-response integration buffer unconditionally and cannot be configured out of it. Compression is a separate class of the same behaviour and can appear at any hop, since a gzip encoder that has not been told to flush per write will happily wait for 16 KB of input before emitting anything.

The consequence for diagnosis is that "which hop" is a real question with more than one candidate, and guessing costs a deploy per guess.

The experiment that names the culprit#

It takes one command to place the fault on one side of a line. Run the request twice with curl -N, once against the origin directly and once through the public hostname, and timestamp each line as it arrives:

curl -N -s -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Explain TCP slow start."}]}' \
  http://10.0.3.12:8080/api/chat \
| while IFS= read -r line; do printf '%s %s\n' "$(date +%s.%N)" "$line"; done

Against the origin, the timestamps should climb steadily:

1717490102.318 event: delta
1717490102.319 data: {"text":"TCP"}
1717490102.361 event: delta
1717490102.362 data: {"text":" slow"}
1717490102.404 event: delta

Against the public hostname, the same command on a buffered path prints nothing for nine seconds and then several hundred lines sharing a timestamp to the millisecond. That single pair of runs settles it: identical application, identical response, different path, different arrival pattern.

It is worth being explicit about why that settles anything, because the same discipline is what makes the rest of the diagnosis tractable. The two runs are a controlled experiment with exactly one variable moved. Same handler, same prompt, same response bytes, same client binary with the same flags — the only thing that differs is which hops the bytes crossed. A difference in arrival pattern can therefore only be attributed to the hops that differ, which are the ones between the public hostname and the origin address. That is what converts "streaming is broken" into a claim about a bounded set of machines, and each of the three possible outcomes names a different set. If the origin drips and the public hostname dumps, the fault is one of the hops between them. If both runs drip, the application is fine and the problem is in the browser client — a fetch reader that awaits the whole body, or a framework that resolves res.text() before rendering. If both runs dump, the fault is in your handler, and the usual cause is building the entire answer before writing anything, or a runtime whose response object needs an explicit flush.

Note that -N on the client side is required for the test to mean anything, since curl buffers its own output when writing to a pipe. Diagnosing a buffering problem with a buffering client has cost people entire afternoons, and the browser has its own version of the same trap: a response whose content type the browser has to sniff is held for the first 512 to 1024 bytes while it decides what it is looking at, so a stream mistakenly served as text/plain will appear to stall for its first few events and then behave normally. Declaring text/event-stream avoids the sniff entirely.

The other half of the measurement belongs in the browser, because the server log and the user's experience are separated by every hop in this chapter and only one of them is the promise you made. performance.getEntriesByType("resource") gives a responseStart for the request, and the gap between that and the first delta your handler renders is the interval this chapter is about. Reporting it as a client-side metric alongside the server's own first-delta timestamp turns "streaming feels broken in production" into a number that either matches or does not, and a persistent divergence between the two is the alarm you want, since a regression here is a configuration change on a path nobody thought was part of the application.

The thirty-second cut-off#

The all-at-once symptom has a neighbour that looks like it and is not: a route that streams correctly for thirty seconds and then stops mid-answer, with no error event, no sentinel, and no exception in the log. The two share nothing except the shape of the complaint, and the columns below are the whole difference.

Serverless adds a wrinkle worth naming before that table is used as a checklist, because a function platform can produce the all-at-once symptom on its own without any proxy involved. Several runtimes were built around a request-response contract where the handler returns a value and the platform sends it, and in that model there is no way to write bytes early — the platform has to have the whole response before it can transmit anything. Streaming from those requires an explicit opt-in with a different handler shape: AWS Lambda needs awslambda.streamifyResponse and a Function URL rather than an API Gateway REST integration, and several hosting platforms distinguish between a Node runtime that supports streaming responses and an older serverless mode that does not. Running the origin test against the function's own URL, bypassing every proxy, is what separates this from the nginx case; if the function itself dumps, no proxy configuration will help.

The limitation there is a shape rather than a missing feature, which is why it cannot be turned on with a flag. A handler that returns a value hands the platform a finished object and the platform sends it — so by construction the platform has the entire response before the first byte can leave, because that is what returning a value means. There is no moment in that contract at which a partial answer exists as far as the platform is concerned, and no setting can create one. Streaming needs a handler that is given something to write into rather than something to return, which is why every opt-in on every platform changes the signature of the function rather than a line of configuration.

The cut-off case deserves one more observation, because it is where this chapter meets cancellation. When the platform kills the function, it does not unwind your stack. There is no catch, no finally, no event: error, no partial persistence, and no abort propagated to the provider — so the model finishes generating all 900 tokens into a socket owned by a process that no longer exists, and you are billed for every one of them while the user has 400 tokens of an answer and no indication that it stopped early. That makes the thirty-second cut-off more expensive than it looks from the user-facing symptom, and it is an argument for a route-level budget — cap max_tokens so that a worst-case generation finishes inside the platform limit — rather than only for raising the limit.

Buffering also degrades the two mechanisms built in the preceding chapters, which is worth knowing before you conclude it is merely a latency annoyance. An event: error emitted at token 400 arrives with everything else at the end, so the client learns about the failure after it has rendered the answer. And a client disconnect is detected on write, so a proxy that is not passing bytes along may not surface the closed socket to your handler for seconds, during which the abort chain has nothing to fire on.

The second of those deserves its own step, because it quietly undoes the previous chapter. A disconnect is noticed when a write fails, and a write fails only when the bytes have nowhere left to go. A buffering proxy is precisely the component that gives them somewhere to go: your write succeeds because the proxy accepted it, and the proxy accepted it whether or not the browser is still on the other side. So every write reports success, nothing raises, and the signal that was supposed to travel upstream is never generated in the first place. For an answer small enough to sit entirely inside the proxy's buffers, that condition holds for the whole generation — the fix from the abort chapter is present, correct, and never invoked.

Not every platform can be fixed by configuration. API gateways that inspect or transform response bodies, WAFs doing content scanning, and some managed function runtimes will buffer regardless of headers, and the answer there is architectural: put the streaming route behind an ingress — the proxy that fronts your cluster and decides which service a request reaches — that supports it, and accept that this is a constraint on where that route can live rather than a setting someone forgot.

So the operating position is that a streaming route has a path, the path has opinions, and neither your code nor your log can see them. Send the headers that ask intermediaries to leave the body alone, configure the hops you own, and keep the two-run curl comparison somewhere it can be re-run, because the arrangement that works today is one edge configuration change away from not working and nothing in your application will report the difference.

With the path clear, tokens reach the browser as they are produced, which is the outcome this whole topic has been arranging. It also means that when a stream dies at token 400, the reader is looking at 400 tokens of it.