Abandoned generations

A stream that behaves is one whose request and generation begin and end together. That is the arrangement local development almost always produces, since a developer watching their own terminal does not close the tab halfway, and it is the arrangement everything in the preceding topic assumed: bytes are framed correctly, a sentinel arrives, a Stop button carries an abort upstream. This topic is about the other arrangement, the one production supplies — a generation started by a request that carries on after the request is over — and about the four independent controls a streaming endpoint needs before it can survive it.

Two nouns have to be pulled apart before any of that can be reasoned about, because everything here turns on their being separate objects. The HTTP request is the browser's question, and it lives exactly as long as the socket carrying it. The generation is the unit of work that answers the question: one model call, running from the moment your server hands the provider a prompt to the moment the decoder emits a stop sequence or hits the max_tokens ceiling, together with everything held on its behalf while it runs — a worker in your process, an upstream TCP connection checked out of a pool, a partially accumulated answer on your heap, and a line on next month's invoice. A generation whose reader has gone while the decoder carries on producing tokens is an abandoned generation — orphaned is the other word teams use — and it is the failure this chapter is about.

Keeping those two lifetimes apart is not a decision anybody made. It follows from there being two connections and nothing joining them. Your process holds one socket to the browser and a second to the provider, and each is an independent conversation with an independent peer. A socket carries no information about any other socket, so the browser's departure reaches your kernel and travels no further than your kernel. The provider, for its part, is not reading anything from you after the prompt: it accepted the request, started a decoder, and is now writing. Nothing arriving on its side of the wire says that the reader beyond your process has gone. Joining the two lifetimes is therefore work your relay has to do deliberately, and a relay that does not do it is not broken — it is the default.

Drawing the line is what makes the remaining chapters possible: once a generation is an object rather than a side effect of one HTTP response, it can be given an id, looked up, cancelled deliberately, written to storage, resumed by a different process, and reconciled against a bill. You meet the failure in the plainest handler in the codebase, a route that opens a streaming request to a provider and copies each delta through to a browser with nothing in it that mentions disconnection at all. That handler passes every test, because the failure raises no exception, writes no log line, and changes nothing any user still watching can see.

How a vanished client is actually noticed#

There are three ways a disconnect reaches your handler, and they differ by roughly three orders of magnitude in how fast they arrive.

The clean case is a browser tab closing, a page navigating away, or a client calling AbortController.abort(). The TCP stack sends a FIN, your kernel marks the socket half-closed, and the runtime turns that into something you can subscribe to: req.signal firing on a Web-standard Request, res.on("close") in Node's HTTP server, an http.disconnect message on an ASGI receive channel, r.Context().Done() closing in Go. This arrives within a round trip, typically under 50 ms on a domestic connection, and it is the case everybody tests, because it is the case you can reproduce by closing a tab on your laptop.

The second case is a client that stops existing without saying so: a laptop lid closes, a phone's radio hands off and the NAT binding is dropped, a train enters a tunnel. No FIN is sent, so nothing arrives at your socket, and no runtime event fires. You discover the loss only by writing, and even then not promptly. Your first few writes succeed, because they only copy bytes into the kernel send buffer, which on Linux starts around 16 KB and can auto-tune upward past 4 MB. Once the buffer fills, the retransmission timer takes over, and Linux's default tcp_retries2 of 15 means the connection is not declared dead for roughly 924 seconds — a little over fifteen minutes — of exponentially backed-off retransmits. A 900-token answer finishes decoding a hundred times over inside that window.

Two steps in that account are worth slowing down, because at each one a reader can reasonably expect an error and does not get one. The first is that a successful write is no evidence of a live peer. A write returns once the bytes have been copied into the kernel's send buffer, and that copy completes before anything has been put on the network at all, so the return value is a statement about your own memory rather than about the far end. The second is where the fifteen minutes comes from. Each retransmission waits roughly twice as long as the one before it, so the budget is not fifteen equal delays but a doubling sequence, dominated by its last two or three attempts; a sum of that shape is a little under twice its largest term, which is how a first retry measured in fractions of a second ends at 924. The decoder, meanwhile, is entirely unaffected, because none of this is happening on the connection it is writing to.

There is a partial defence, and it is worth knowing its limits. Writing something small and harmless at a fixed interval — an SSE comment line, : ping, every ten or fifteen seconds — guarantees that the socket is exercised even when the model is quiet, which is what turns a silent peer into a failed write eventually rather than never. During a dense stream of deltas at 45 tokens per second the answer is already its own heartbeat, so the interval only earns its keep in the gaps: a tool call being executed, a reasoning phase before the first visible token, a queue wait upstream. What it does not do is shorten the fifteen minutes, because the retransmission budget is the same whatever bytes fill it. Detecting a silent client faster than that requires an acknowledgement travelling the other way, which the reader's cursor two chapters from now supplies as a side effect of something else.

The third case is the one that catches teams who tested the first two. If there is a proxy, an ingress controller, or a CDN — a content delivery network, the tier of caching servers a request passes through before it reaches you — terminating TCP between the browser and your process, then the socket your handler holds belongs to the intermediary. When the browser vanishes, the intermediary may close its upstream connection to you immediately, or it may hold it until its own idle timeout — commonly 60 seconds — or it may hold it for the life of the response so it can finish caching a body you never asked it to cache. What your handler observes is the intermediary's behaviour, and that behaviour is a configuration setting on a box you may not own. The last chapter of this topic takes that hop chain apart properly; for now it is enough that "the client disconnected" is a claim about the nearest hop.

The belief that the diagram contradicts is worth naming, because it is nearly universal and it is held by people who know perfectly well how TCP works. Closing the tab feels like cancelling the work, and it feels that way because everything else on the web behaves as though it does: a half-loaded image stops loading, a video stops buffering, a page navigating away stops fetching. All of those stop because the browser was the thing doing the work and the browser has gone. Here the browser was never doing the work. The decoder is running in somebody else's data centre, it was started by your server, and the only participant in a position to stop it is the one holding the second socket.

Cancellation is an action, not a consequence#

Once you know the client is gone, stopping the generation is a separate thing you have to do. For a streaming completion there is no cancel endpoint to call at the major providers — the cancel APIs that exist cover asynchronous work, like Anthropic's Batches API or a background response object, and none of them applies to a request whose tokens are arriving over an open socket. The way you cancel a streaming completion is by destroying the HTTP connection carrying it. The provider observes the transport close on its side and stops the decoder.

In practice that means threading the abort signal from the inbound request to the outbound one, which is the propagation the preceding topic worked through hop by hop. What is worth restating here is not the mechanism but its shape: the naive relay is correct in every test, and the difference between it and the propagating one is a single property that no user-visible behaviour depends on.

// Naive: the client's disconnect never reaches the provider.
export async function POST(req: Request): Promise<Response> {
  const upstream = await fetch(PROVIDER_URL, {
    method: "POST",
    headers: providerHeaders,
    body: JSON.stringify({ ...params, stream: true }),
  });
  return new Response(upstream.body, {
    headers: { "content-type": "text/event-stream" },
  });
}

// Propagating: aborting req tears down the provider connection too.
export async function POST(req: Request): Promise<Response> {
  const upstream = await fetch(PROVIDER_URL, {
    method: "POST",
    headers: providerHeaders,
    body: JSON.stringify({ ...params, stream: true }),
    signal: req.signal,
  });

  req.signal.addEventListener("abort", () => {
    metrics.increment("generation.cancelled", {
      reason: "client_disconnect",
    });
  });

  return new Response(upstream.body, {
    headers: { "content-type": "text/event-stream" },
  });
}

One property is worth stating precisely, because teams assume the opposite: cancelling saves the tokens not yet produced, and nothing else. You are billed for every output token the decoder emitted before the connection died, plus the input tokens of the whole prompt, which were charged the moment the request was accepted. Aborting at token 84 of a 900-token answer is a 91% saving on output for that request, and a 0% saving on input — which matters more than it sounds when the prompt carries 12,000 tokens of retrieved context and the answer is 900. There is also slack at the end: a few tokens are typically in flight or already counted when the reset arrives, so cancelled requests bill slightly above the last token you saw. Do not build an accounting check that treats those as a discrepancy. An aborted stream also loses the final usage event entirely, since that event is written after the text, so the only figure you have for a cancelled request is one you compute yourself with a local tokeniser.

The two percentages in that paragraph are worth deriving rather than accepting, because they come from different denominators and the difference is the whole point. Output is charged per token produced, so aborting at 84 leaves everything between 84 and 900 unproduced; that difference divided by 900 is a little over ninety per cent, which is where the 91% comes from. Input is charged once, for the whole prompt, at the moment the request is accepted, and no later event can reduce a charge that has already been settled — that is the 0%. The two do not average into anything meaningful, because they are shares of two quantities of very different size. A prompt of 12,000 tokens against a 900-token answer is more than thirteen times as many tokens, so saving nine tenths of the smaller side leaves most of the request's tokens paid for regardless.

What an abandoned generation is holding#

The invoice is the visible cost and usually not the largest one. Take a service with 40,000 generations a day, answers averaging 900 output tokens, decoding at about 45 tokens per second, and a measured 15% of clients that disconnect before the last token — a figure that is unremarkable for a mobile-heavy consumer product and low for one with long answers.

The token waste is arithmetic. If the median abandonment lands at token 90, each abandoned generation produces roughly 810 tokens nobody reads. At 6,000 abandoned generations a day that is 4.86 million wasted output tokens daily, and at a frontier-model list price of $15 per million output tokens, about $73 a day, or $2,200 a month. That is real and it is also the number teams fixate on.

Every step in that chain is worth checking, because it is short enough that one wrong figure in the middle survives to the end unchallenged. Fifteen per cent of 40,000 generations is 6,000, and that is a count of generations rather than a rate. A 900-token answer abandoned at token 90 leaves the difference between the two, which is 810; the 90 already delivered are not waste, because somebody read them. Multiplying a per-generation quantity by a count of generations is legitimate here only because both are daily figures, and 6,000 lots of 810 is 4.86 million tokens a day. Dividing by a million and multiplying by the $15 rate gives roughly $73, and thirty of those is about $2,200. Nothing in the chain depends on when in the day an abandonment happened, which is why a median is enough and a full distribution buys nothing.

The capacity cost is less obvious. Each of those 810-token tails takes about 18 seconds of decode, which is 18 seconds during which one of your workers is parked in a read loop, one upstream TCP connection plus its TLS session is checked out of the connection pool, and the partially accumulated answer is sitting in your heap. Six thousand of those per day is 30 hours of aggregate zombie decode time — a zombie being the ordinary name for an abandoned generation that is still decoding — concentrated in the same peak hours as the live traffic they are competing with. If your HTTP client pools connections per origin — undici defaults to 128 connections per origin, Go's http.Transport to 100 idle per host — then a few hundred concurrent zombies will exhaust the pool and new, live requests queue behind them, which shows up as time-to-first-token regressions for users who did nothing wrong. Your provider rate limit is charged the same way: output tokens per minute is consumed by tokens you discard exactly as it is by tokens you render, so at the limit an abandonment rate of 15% is a 15% cut in the throughput available to real users.

Two of those figures are the same 810 tokens read a different way, and one of them hides a mechanism. At about 45 tokens per second, 810 tokens is 18 seconds of decode, and 18 seconds is how long each of the resources above stays committed. Six thousand abandonments at 18 seconds apiece is a hundred and eight thousand seconds, which is the 30 hours — an aggregate spread across whatever concurrency the fleet is running, not a wall-clock stretch during which nothing else happens. The pool exhaustion then follows from a property of HTTP clients that is easy to forget: a pooled connection is checked out for the duration of the response body, not the duration of the request, so a zombie holds one for the entire 18 seconds of tokens nobody wants. When every connection to an origin is checked out, the next request does not fail. It waits, and a wait before the first byte is precisely what time-to-first-token measures. The rate limit gives way for a related reason: the meter runs at the provider, counting what its decoder emitted, and the discarding happens in your process where the provider cannot see it.

On serverless the shape changes but does not improve. A 20-second zombie on a 1 GB function is 20 GB-seconds, about $0.00033 at typical pricing, which is negligible next to the tokens; the binding constraint there is the concurrency limit, since an invocation held open for 20 seconds after its reader left is an invocation slot unavailable for the next request.

Timeouts, leaks, and generations worth finishing#

An abandoned generation is not a timeout problem, and reaching for a request timeout to solve it makes things worse: a timeout tight enough to reap zombies at 20 seconds also kills legitimate long answers, and one loose enough to spare them reaps nothing. The two conditions are independent, which is why they need independent mechanisms.

It is also not a memory leak, even though a heap graph produced by a fleet full of zombies looks exactly like one — steady growth through the day, a clean reset on restart. A later chapter separates those two properly, because the fix for one does nothing for the other.

Most importantly, it is not automatically a bug. Everything above describes waste on the assumption that the abandoned output has no value, and that assumption holds for a code assistant where the user who closed the tab will simply ask again. It does not hold for a ninety-second compliance report that gets exported to PDF, where killing the generation the instant a laptop sleeps destroys work the user already paid for and will pay for again.

So the position to hold at the end of this chapter is that a generation is an object with a lifetime, that the lifetime is not the request's, and that the difference is measurable before it is fixable — billed output tokens over delivered output tokens, per route, is an afternoon of work and tells you whether you have a problem worth spending a week on. Detection of the client's departure is available in three forms of wildly different quality, and only the fastest of them is the one you will test with.

Now that a disconnect is something your server can observe, and cancellation is something it can issue, the question is which of the two available responses you owe each caller.