Streaming costs the consumer that cannot spend it

Time to first token is the median wait, across requests, before any byte of the answer arrives, and it is the number that streaming improves and the number most streaming proposals are written around. It is a real measurement and it is about the arrival of bytes rather than about the completion of work, which makes it the wrong summary for any consumer that cannot start work on a byte. The question that decides whether a stream is worth building is what its consumer does at 210 milliseconds that it could not do at 3,400, and for a worker whose next line is queue.send(obj), the answer is that it waits.

This chapter is about that decision and about the design that follows when the answer is genuinely mixed, because the interesting case is not a worker on its own — it is an object that feeds a browser and a worker at once, where the instinct to have both read the same stream produces two incremental parsers that disagree.

What the worker bought and what it paid#

Streaming to a consumer that acts only on the completed object is a strictly additive change to the failure surface. The value arrives at the same time, the token bill is identical because you pay per token generated either way, and everything else on the ledger moves in one direction.

The bottom row is the one that stops the argument that streaming is at least free. The connection is open for the same duration in both cases, so streaming does not reduce the concurrency cost of a slow generation; it only changes whether bytes move during the wait. The row above it is where the incidents come from, because a streaming response has already committed to HTTP 200 by the time anything goes wrong, so every dashboard, alert and service level objective built on status codes reports success on the failures that matter. The StreamOutcome record built earlier in this topic exists to replace that lost signal, and it is work the non-streaming version never needed.

That row repays one level down, because "a truncated stream is a 200" sounds like a quirk and is in fact forced by the protocol. HTTP requires the status line to be sent ahead of the body, and a streaming response begins sending body bytes as soon as it has any — which is the whole of what streaming means. The status is therefore committed at the instant the first chunk leaves, before the generation has finished and so before anything can be known about whether it will finish at all. No mechanism exists by which a stream that dies at eighty per cent retracts its 200; the bytes carrying it left seconds earlier. Everything downstream that reads status codes — the load balancer's metrics, the alerting, the service level objective, a generated client's error handling — is reading a value that was fixed before the outcome existed. Seen that way, StreamOutcome is not extra telemetry somebody chose to add. It is the replacement for a signal the design traded away in exchange for the first byte.

Two consumers that can actually spend it#

There are backend consumers for which the answer is not "it waits", and refusing to name them turns a good rule into a slogan.

The first is a consumer that starts real work on a field before the object closes. If an enrichment result opens with customer_id, a worker can close that field at 400 ms and immediately fire the account lookup and the document fetch that the next stage needs, so that when the object closes at 3,400 ms those round trips are already done and the stage finishes 250 ms sooner. The same shape appears with a moderation or policy gate that can abort the generation the moment a disqualifying field arrives, saving the remaining 2,800 ms of tokens on the fraction of requests that trip it. Both are subject to every render-gate rule in this topic — the field must be syntactically terminated and semantically self-contained, and the work started must be revocable, which is why prefetching is a good fit and issuing a refund is not.

Both savings are bounded, and knowing what bounds them is what lets you decide before building rather than after. The prefetch saves the smaller of two quantities: how long the lookups take, and how much room there is between the field closing and the object closing. Here the field closes at 400 ms and the object at 3,400, so the window is three seconds and the lookups are 250 milliseconds — the lookups hide inside the window entirely, and 250 milliseconds is the whole prize. Make the lookups slower and the saving stops growing once they outlast the window, because the object arrives before they finish. The abort saves the tokens that would have been generated after the disqualifying field, which is the remaining 2,800 milliseconds of a 3,400-millisecond generation, multiplied by the share of requests that actually trip the gate. That multiplication is the part usually skipped, and it is the part that decides whether a gate firing on one request in a thousand is worth an incremental parser in a worker.

The second is liveness. A load balancer or content delivery network in front of your service usually has an idle timeout measured in tens of seconds — 60 seconds on an AWS Application Load Balancer (ALB) by default, 100 on Cloudflare's proxy — and a non-streaming call that takes four minutes to generate a large structured document sends no bytes for four minutes and gets killed. Streaming keeps bytes moving, which turns an impossible request into a possible one. This is a real reason to stream to a backend, and it is worth recognising as an infrastructure workaround rather than a product benefit, because the same effect is available from a job-plus-poll design that does not put an incremental parser in the worker.

When both consumers are real#

The instinct when an object feeds a browser and a worker is to have both of them consume the same stream, because the stream is already there. That gives you two independent incremental parsers, in two languages, with two repair policies and two definitions of complete, and they will disagree. The disagreement is not hypothetical: the browser's parser drops unterminated numbers and the worker's keeps them, or the browser treats a heartbeat timeout as withdrawal while the worker is still reading, and now the customer has seen a number that the record does not contain.

One component owns the parser, and it is the relay — the endpoint of your own that sits between the provider and everything else. It consumes the provider's delta protocol, maintains the lexer state across chunk boundaries, emits snapshots to the browser at a coalesced rate, and hands the worker exactly one thing: the closed, validated object.

The failure path is what makes the design worth its cost. When the stream dies at 80%, the relay emits result_end with a truncation status to the browser, which withdraws, and sends nothing at all to the queue, so there is no partial record to reconcile later. A single component decides once what happened, and both consumers inherit that decision.

async def relay(request, provider_stream, publisher):
    parser, sent_at = PartialJson(), 0.0
    async for delta in provider_stream:
        parser.feed(delta.partial_json.encode("utf-8"))
        now = time.monotonic()
        if now - sent_at > 0.066:                     # ~15 Hz, coalesced
            await request.send_sse("snapshot", parser.snapshot())
            sent_at = now

    outcome = classify(...)                           # the three truncation causes
    if outcome.cause is not None:
        await request.send_sse("result_end", {"status": "truncated", "cause": outcome.cause})
        return                                        # nothing is published

    value = Enrichment.model_validate(parser.snapshot())
    await request.send_sse("result_end", {"status": "ok"})
    await publisher.send(value.model_dump())          # the worker's only input

Two properties of that handler are load-bearing. The publish happens after validation and only after it, so the queue's contract is "a valid object or no message". And the browser's result_end is sent before the publish rather than after, so a queue outage cannot leave the user staring at a provisional panel forever — though it does mean a committed render can exist alongside a failed publish, which is a trade you should make deliberately and cover with a retry on the publisher rather than by reordering the two lines.

The 0.066 is the reciprocal of the rate in the comment beside it: fifteen updates a second is one every sixty-six milliseconds or so. It is compared against a monotonic clock rather than wall time, so that a clock adjustment cannot make the measured interval negative and stall the sends. Coalescing here rather than in the browser is what makes the update rate a property of the system instead of a property of each client, which starts to matter the moment there is more than one kind of client — and by then the relay is also the only component that knows what a complete value is.

The worker's code, meanwhile, is the code it had before any of this started: receive a message, trust that it is a whole object, validate it defensively if you like, act.

The infrastructure the stream drags along#

Deciding to stream commits you to a set of operational details that a request/response call never touches, and they are worth pricing before the decision rather than after.

Buffering proxies are the classic one, and topic 2.9 covers the mechanism and the directives that stop it. The point that belongs here is the shape of the failure: nginx defaults to proxy_buffering on, which will happily accumulate an entire SSE response and deliver it in one piece, producing a service that behaves exactly like the non-streaming version in production while streaming perfectly on the developer's laptop. Response compression can do the same thing at a different layer, because a compressor works on blocks and will hold bytes back until it has a block worth emitting unless it is told to flush on every write. Both are configuration you have to know to look for, and both fail in the direction that hides the problem — which means a streaming migration can be complete, deployed and delivering none of the benefit it was justified by, without a single alert firing.

Observability changes shape. The interesting metrics are no longer status codes and total latency but the distribution of StreamOutcome.cause, the inter-chunk gap p99, the rate of snapshots that fail the monotonic assertion, and time-to-first-useful-pixel rather than time-to-first-token. Usage accounting moves too: token counts arrive in the terminal event, so a client that stops reading after it has what it wants never learns what the request cost.

Cancellation becomes something you have to implement. A non-streaming call that the user abandons is a wasted generation you find out about at the end; a streaming one gives you the chance to abort the upstream when the downstream closes, and that is the single largest saving available on interactive workloads with impatient users. The opportunity exists for a mechanical reason worth knowing: a client disconnect is observable to a handler that is already writing to a socket and invisible to one that is not. Code holding a whole response in memory has nothing to write and therefore nothing to fail on, so it learns of the abandonment only when it tries to deliver a result to nobody. Streaming turns that into an event at the moment it occurs, and it is the one item on this ledger that moves in the operator's favour without an incremental parser attached to it.

Where the argument stops#

It is not a case against streaming. Where a person is reading the output, the difference between a first sentence at 280 ms and a blank panel for 3.4 seconds is the difference between a tool that feels alive and one that feels broken, and everything in this topic exists so that experience can be built without lying to the user. The claim is narrower: a consumer that cannot act on partial output is paying for someone else's benefit.

It is also not a question about transport. SSE, WebSockets, HTTP/2 streams and chunked responses all give you framing and none of them decide who the stream is for; the framing guarantee is about the envelope and the value layer is yours either way.

And it is not an argument for putting the decision off. The parser, the render policy, the terminal event and the identity scheme are cheap to design before the first client and expensive to retrofit, because by then two consumers have independent opinions about what a complete value is. The consumer question comes first precisely because it is the one that can delete the other six.

What is left is a sequence of decisions to make before the first line of client code. Establish which of the two protocols the provider speaks and at which layer you parse. Decide whether you relay and re-frame, and if you do, put the incremental parser there and nowhere else. Write the boundary-aware parser or adopt one, and hold it to the every-cut-point test. Write the render policy field by field, including the fields you refuse to stream, and keep it where a reviewer can find it. Define the terminal event, the heartbeat, and the three states with one exit each. Assign identity at the producer, assert that updates only add information, and reserve the slots from the schema. Then, before shipping any of it, name the consumer who is going to spend the milliseconds you are buying, and if there is not one, stop.