Retrying a partially delivered answer

Retrying a failed call is one of the best-established patterns in backend work, and it is correct for the call this used to be. A non-streamed chat completion is atomic from the caller's perspective: either you have a response object or you have an exception, and if you have an exception nothing has been delivered anywhere, so re-issuing is invisible and safe. Teams carry that instinct forward when they turn on streaming, and the code barely changes.

What changed is that part of the response has now been delivered to a human, is being read, and cannot be recalled. A retry is no longer a network operation; it is an edit to a document someone is looking at. This chapter is about where the boundary sits, what the two available continuation mechanisms cost, and how to store the result so that a turn with two attempts behind it has exactly one answer.

The premise underneath retry is never written down, which is why it survives a change that invalidates it. Retry is safe when the failed attempt left no trace anybody can observe, because that is what makes a second attempt indistinguishable from a first. A non-streamed call satisfies the premise exactly: the only thing it was going to produce was a response object you never received, nothing else in the world moved, and re-issuing restores the situation to what it was beforehand. A streamed call satisfies it up to the first delta and never again. That delta was rendered, a person read it, and the words are in their head and possibly in their notes. No operation available to your code un-renders them. The call is still a network operation; the system it is operating on now contains a reader, and the reader has state.

The boundary the first delta draws#

If zero deltas have reached the client, retrying is exactly as safe as it was before streaming and should be automatic, silent, and capped. This is also the common case: authentication failures, rate limits, model-not-found, and most capacity errors surface before or at the very start of the stream, so silent retry before the first delta removes the large majority of user-visible errors on its own.

Why the safe side of the boundary covers so much is worth one step, because it is what makes the cheap half of this chapter the valuable half. Authentication, rate limits, an unknown model name and most capacity rejections are decided at admission: before any prompt is prefilled, and long before any token is decoded. A provider answers them with a status code while nothing has begun. A failure decided before work starts cannot occur after work has been delivered, which is why those errors land in the zero-deltas case by construction rather than by luck. The failures that reach a reader mid-answer are a different and much smaller population — an upstream degrading part-way through, a path breaking, a process dying — and they are the ones the rest of the chapter is for.

Capped matters as much as silent. A single automatic retry before first delta, with 200 to 400 ms of backoff and full jitter — a random delay drawn from the whole interval rather than a fixed one — recovers essentially all transient failures; two adds little; unbounded retries against an overloaded provider are how one team's outage becomes that provider's outage, because a provider that is overloaded is overloaded for everyone and an unbacked-off retry across a thousand clients is a load amplifier pointed at a system already in trouble. Cap attempts per turn at two and per session at something small, and count exhausted budgets as a metric, because a rising retry rate is an early signal of upstream trouble that arrives before the error rate does.

The jitter is doing something specific, and it is worth one step because a fixed delay looks like the safe, predictable choice and is the one that hurts. An upstream failure is a single event: the provider sheds load, or a shard goes, and every client in flight at that instant is rejected at that instant. A fixed backoff adds the same number of milliseconds to every one of those clients, so the wave that arrived together leaves together and arrives together again, one delay later, against a system that has had exactly that delay to recover. Drawing the delay at random from the whole interval instead maps that single instant onto a spread of instants, and the load the provider sees on the way back up is smeared across the window rather than stacked at the end of it. Halving the delay would not have helped and doubling it would not have helped, because the problem was never the length of the wait. It was that everybody waited the same amount.

Past the first delta, none of that applies, and the two available mechanisms fail differently.

The two ways to continue, and what each costs#

Those output figures are worth arriving at rather than reading off, because the comparison between them is the only part of this decision that has a number in it. The answer wanted about 900 tokens and the failure came at about 400, and those 400 were generated on the provider's hardware, so they were billed whatever happened next — a failure part-way through a stream does not refund the tokens that already left. Re-issuing generates the whole answer again, so the account is charged the 400 that were thrown away plus a fresh 900, which is roughly 1,300 output tokens for an answer 900 tokens long. Continuing generates only what is missing: 900 wanted, less the 400 already delivered, leaves about 500, and the 400 plus that 500 comes back to the 900 a clean run would have cost. The two input columns move the other way. Re-issuing sends the original prompt a second time and nothing else, while continuing sends that same prompt plus the delivered text as a seeded assistant turn — so the delivered 400 are paid for twice, once as output when they were generated and again as input when they are handed back.

The latency rows are one rate seen twice, and checking that is what makes them trustworthy rather than plausible. A 900-token answer in about nine seconds is a hundred tokens a second, and the remaining 500 tokens at a hundred a second is about five seconds. Nothing in the continuation is faster than the original; it is shorter, and the whole of the saving is the part that does not have to be generated again.

Re-issuing is the simpler mechanism and the more expensive one, and its real cost is not on that invoice. The answer that comes back is a different answer, because a nonzero temperature — the sampling setting that lets the model choose among plausible next tokens rather than always taking the likeliest — makes two runs of the same prompt diverge. On anything with figures, citations, or clause references, the reader who has already absorbed the first version now has to be told that what they read is being replaced.

The obvious repair is to set the temperature to zero and re-issue, on the reasoning that a deterministic model gives back the same answer and the reader never notices the seam. It is worth taking seriously, because it removes one of the two problems and leaves the other exactly where it was. Suppose it worked perfectly and the second generation were identical to the first, token for token. The second generation still starts at the beginning. The reader has already been shown the first four hundred tokens of that text, and unless the client clears the surface those tokens arrive on the screen a second time — which is precisely the doubled opening in the summary reproduced later in this chapter. Determinism turns a contradiction into a duplication. It does not turn either of them into nothing, because the thing that has to be undone is not the divergence — it is that a prefix was rendered, and no sampling setting reaches a screen. The divergence is what makes an unmarked re-issue dangerous; the restart is what makes it visible, and only the second of those is under the API's control.

Continuing uses assistant-turn prefill, where you send the original messages and seed the assistant's reply with the tokens already delivered so the model carries on from them. It is cheaper and faster, and it preserves what the reader has already read, which is a genuine advantage. The catch is that this is a different prompt producing a different task — completion of an existing draft rather than generation of a new one — and models are noticeably worse at it on structured documents. In practice, a summary cut off inside section 2 comes back having restarted at "1.", or having re-stated the sentence it was cut off in, or with a heading level that does not match. Roughly speaking, prose continues acceptably and anything with numbering, JSON structure, or a table does not.

Why the damage concentrates in exactly those places is worth following, because it predicts which of your endpoints will survive continuation and which will not. A model continuing a prefill is doing the only thing it ever does: reading everything in front of it and choosing what comes next. What is in front of it is a prompt asking for a summary, followed by a stretch of text that reads like the opening of a summary. That is an ambiguous document. One reading is this answer is under way, carry on from the middle; another is this is what an answer to that request looks like, produce one. Nothing in the seeded text disambiguates them, because a partial document and a whole document look the same until the point where they differ. The tokens that carry the strongest signal about position — a heading, a list number, an opening brace — are also the tokens the model has seen most often at the start of a document, so on exactly those tokens the second reading wins. That is the mechanism behind a summary cut off in section 2 coming back at "1.", and it is why the failure is not spread evenly through the continuation but clustered on its structural markers. Prose has no such markers, which is the whole of why prose continues acceptably.

The practical form of that is a test you can run on your own output rather than a rule to remember. Take a finished answer from the endpoint, cut it at several points, feed each prefix back as a seeded assistant turn, and read the joins. An endpoint whose answers carry no numbering and no enclosing syntax will join cleanly almost every time. An endpoint that emits JSON will fail on nearly every cut, and it will fail in the same place each time, which is the signature of a mechanism rather than of bad luck.

The turn state an interrupted answer needs#

The partial text needs two properties: it must be distinguishable from a finished answer, and it must never be silently overwritten or silently extended. The completion signal from earlier in this topic — the stream is complete only if a terminal sentinel arrived — is what supplies the first.

type Turn =
  | { state: "streaming"; text: string }
  | { state: "complete"; text: string }
  | { state: "interrupted"; text: string; reason: string };

function onStreamEnd(turn: Turn, sawSentinel: boolean, err?: StreamError) {
  if (sawSentinel) return { state: "complete", text: turn.text } as const;
  return {
    state: "interrupted",
    text: turn.text,
    reason: err?.code ?? "connection_lost",
  } as const;
}

An interrupted turn renders with the text still visible, dimmed or rule-separated, labelled as incomplete, and with an explicit control offering to continue or to regenerate. Both mechanisms from the table above are then a user action rather than a silent substitution, and the interface has said which one it is about to perform.

Storing two attempts at one answer#

Persistence needs the same discipline, and getting it wrong is how the defect outlives the session it happened in. The relay persists the partial with a partial status, and a retry must not turn that into two assistant messages on the same turn.

CREATE TABLE generation (
  id            uuid PRIMARY KEY,
  turn_id       uuid NOT NULL REFERENCES turn(id),
  attempt       int  NOT NULL,
  status        text NOT NULL,   -- complete | partial | cancelled
  strategy      text,            -- reissue | continue
  text          text NOT NULL,
  canonical     boolean NOT NULL DEFAULT false
);

CREATE UNIQUE INDEX one_canonical_per_turn
  ON generation (turn_id) WHERE canonical;

The partial index is the part that does the work — partial meaning that it covers only the rows matching its WHERE clause, so uniqueness is enforced across the canonical rows alone: the database refuses to hold two canonical generations for one turn, so the question "which of these is the answer" has exactly one answer at all times and cannot drift as code paths multiply. A retry inserts a second row with attempt = 2, and promoting it to canonical demotes the first in the same transaction.

Putting that rule in the database rather than in the code that writes the rows is not fastidiousness, and the reason is the same reason this chapter exists at all. The situation that produces a second generation is by definition a situation where something went wrong at an unplanned moment, so the code paths that can reach the promotion are the retry handler, the user pressing the control a second time, a background job reconciling an interrupted turn, and whatever gets written next year. Every one of them can be correct on its own and still overlap with another, because two of them can be running at the same instant and neither can see the other's uncommitted work. A check-then-write in application code is two operations with a gap between them, and the gap is where the second writer fits. A unique index has no gap: the database evaluates it as part of the write, so the second promotion fails rather than succeeding quietly. What that buys is not tidiness. It is that the question "which of these is the answer" cannot develop a second answer while nobody is looking.

That structure decides what the next turn sends. The history replayed on the following request should contain the canonical generation only, so a reissue retry sends the second attempt and the truncated first attempt never appears; without the constraint, history contains both, and the model reads a conversation in which it apparently answered twice, which it will occasionally remark on. For a continue retry the canonical text is the concatenation of the delivered partial and the continuation, stored as one row rather than two, because that is what the reader saw.

The turn id is also the natural idempotency key, the value that makes a repeated request resolve to the work already done rather than starting a second copy of it. A double-clicked send button, a client that retries on a network blip before any bytes arrive, and a user reloading a page mid-generation all produce a second request for the same turn, and keying on the turn means they resolve to the existing generation rather than starting a paid second one. Providers offer their own idempotency keys for the same purpose on the upstream leg, and using both closes the gap on either side of your relay.

The turn rather than the request is the right thing to key on, and the distinction is the one that decides whether the key works at all. Idempotency is a property of an effect, not of a message: what must not happen twice is the generation, and the generation is identified by which turn of which conversation it answers. A key derived from the request — a hash of the body, an identifier the client minted — treats two messages as the same job only when the messages match, and the three cases above do not produce matching messages. A double-clicked button sends two requests a few hundred milliseconds apart, a reloaded page sends one with a fresh connection and quite possibly a fresh client-side id, and a network-blip retry may carry a retry counter. All three are one piece of work described three ways. Keying on the work is what makes them collapse.

Where continuation runs out#

Neither mechanism is resumption. There is no provider mechanism that picks up a generation where it stopped with the same sampling state — the decode state is gone the moment the request ends — so continuation is an approximation built out of a second inference call, and it is fair to say that resumable generation is genuinely unsolved at the API level. What can be resumed is delivery of text your own system already holds, which is a different problem with a different solution and belongs to the topic on running streams in production.

The reason that mechanism does not exist is worth one step, because "surely the provider could just carry on" is the first thing anybody asks and it sounds like a feature request rather than an impossibility. A generation in progress is not only the text you have seen. It is a body of working state built up over the prompt and over every token produced so far, held in memory that was allocated to that request when it was admitted. That memory is the scarce resource the whole service is scheduled around, so it is released the moment the request ends, and the request ended — that is what the failure was. There is nothing left to point at, and no name by which you could point at it if there were. A second call therefore has to rebuild its position from the only thing that survived, which is the text, and rebuilding a position from the text is precisely the ambiguous-document problem above. Continuation is not a weaker version of resumption. It is a different operation that happens to produce output in the same place.

There is one case where a second answer may be shown as though it were the first, and it is narrow: nothing had been rendered when the failure occurred. Everything else requires that the reader be told the surface changed — by clearing it visibly before the replacement streams in, or by appending after a marked seam. The narrowness is the point of the rule; a boundary that admits "well, only two sentences had rendered" is a boundary that will be argued down to zero over a year of incremental product decisions.

Structured output deserves one more warning, because it fails in the direction that is hardest to notice. A truncated JSON document continued by prefill will frequently re-open a brace or repeat a key, and the parse either fails loudly, which is fine, or succeeds with a duplicated key whose last value wins, which is not. Where the output is machine-read rather than human-read, re-issue rather than continue, and validate the assembled result before it is treated as canonical.

That is the whole shape of a streaming endpoint you would be willing to defend. It parses events rather than chunks, so nothing arrives halved. It knows what it cost, because it reads the usage-bearing event instead of the deltas. It emits a sentinel, so completion is a positive fact rather than an absence, and it emits an error event when it can while treating silence as the normal failure. It carries the client's abort into the upstream request, and it has a query that proves the abort landed. It runs behind a path that does not buffer, verified with two curl -N runs rather than assumed. And when it dies halfway through, it says so, and what happens next is a decision the reader makes rather than one the client makes on their behalf.

The way to hold onto any of this is to build one. Take a route you already have, stream it end to end, then break it on purpose: kill the upstream mid-generation and watch what the client does, put a default nginx in front of it and time the first byte, click Stop and run the distribution query against your own data. Every failure in this topic is one you can reproduce in an afternoon, and the ones you have reproduced are the ones you will recognise at 03:00.