The durable chunk log

The detach policy from the previous chapter leaves a generation running with no socket attached to it, which means the output has to go somewhere. The obvious somewhere is memory: accumulate deltas into an array, and write the assembled answer to a database when the stream finishes. That works in staging and in every test, and it fails on the first deploy that lands mid-generation, because the array was the answer and the answer lived in one process.

This chapter replaces it with a chunk log: an append-only record in which every chunk the generation emits is written durably at the moment it is produced, identified by the generation id and by a sequence number that is monotonic within the generation — it only ever increases, so 412 is always the position after 411 and never a tie. The generation id names the log, the sequence number names a position in it, and the client's view of the answer is a fold over the entries from 1 to whatever it has seen. That reframing buys three things at once. A pod death loses only the chunks that were in flight at that instant rather than the whole answer. A reconnecting client can be served from the same log a live client is reading, so there is one code path rather than a running-versus-finished fork. And the answer becomes inspectable after the fact: when a user says the output was truncated, the log tells you the last sequence written, its timestamp, and whether a terminal record ever arrived.

What the record has to contain#

The minimum is smaller than people expect.

CREATE TABLE generation_chunk (
  generation_id  uuid    NOT NULL,
  seq            integer NOT NULL,
  kind           text    NOT NULL,  -- 'delta' | 'tool_call' | 'error' | 'done'
  payload        text    NOT NULL,  -- the SSE data line, verbatim
  created_at     timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (generation_id, seq)
);

CREATE TABLE generation (
  id             uuid PRIMARY KEY,
  status         text NOT NULL,     -- 'running' | 'complete' | 'failed'
  last_seq       integer NOT NULL DEFAULT 0,
  finish_reason  text,
  usage          jsonb,
  lease_until    timestamptz,       -- owner heartbeat; NULL once terminal
  created_at     timestamptz NOT NULL DEFAULT now()
);

The composite primary key on (generation_id, seq) is doing real work. It makes the write idempotent, so a retried insert after a network blip on the database connection collides rather than duplicating, and it gives you the replay query for free as a range scan. Storing the payload verbatim as the SSE data line, rather than as parsed text, means the replay path does not have to reconstruct anything: whatever the live client received byte for byte is what a resuming client receives.

Idempotence there deserves one more step, because "the primary key makes the write idempotent" is the kind of sentence that gets nodded at. A retry is only safe if the retried write carries the same key as the original, and a key the storage layer generates cannot have that property: the second attempt arrives with a fresh identity and is stored as a fresh row. Here the key is computed by the writer before the first attempt out of two things it already holds — the generation id, and a sequence number it assigned itself. Both are identical on every attempt, so the second insert collides with the first and is refused rather than appended. The database is not deciding anything about duplication; it is enforcing a decision the writer made before it ever opened a connection.

lease_until is the field teams add second, after the first time a process dies holding a running generation. The owning task heartbeats it every few seconds; a sweeper marks any running generation whose lease expired more than thirty seconds ago as failed. Without it, "running" and "the process that was running this is dead" are indistinguishable, and a client that reconnects to a dead generation waits forever on a stream that will never produce another token.

A heartbeat is the only construction that works there, and the reason is that a dead process cannot report its own death. Every other candidate signal — a status column set on exit, a shutdown hook, a finally block — is written by the process itself, and a process killed by an out-of-memory event or a node drain writes nothing at all. What survives such a death is the absence of something the process would otherwise have done, and an absence is only observable once somebody has said how often the thing was expected. lease_until is that statement. The owner promises to push the timestamp forward every few seconds, so a timestamp that has not moved for thirty is evidence in a way that no field the owner sets could ever be. The persistence of the chunks and the liveness of their owner are separate facts about a generation, and a design that captures the first and not the second is what leaves a status column reading running all night.

Write first, forward second#

The ordering of the durable write and the flush to the client is the part that is easy to get backwards, and the two orders fail in ways that are not symmetric.

Writing first means the log is always a superset of what the client received. If the process dies between the write and the flush, the log contains a chunk that never reached anyone, and the recovery is a replay that sends it again — the client may see something it already had, which is a duplication problem, and duplication is detectable and fixable because both sides can compare sequence numbers.

Flushing first means the log is a subset of what the client received. If the process dies between the flush and the write, the client has rendered chunk 412 and the log ends at 411. Now the reconnecting client asks to continue from 412 and the server has no record of it, no record of anything after it, and no way to tell whether the gap is one chunk or forty. The provider stream is gone with the process, so the missing tokens cannot be regenerated — resampling the same prompt gives you different text, not the continuation. Lost data is unrecoverable in a way that duplicated data never is.

Stating the two orders as a relation between sets makes the asymmetry mechanical rather than a matter of judgement. Call the set of chunks written durably W and the set the client has received D. Writing first means a chunk enters W before it can enter D, so D is always contained in W: whatever the reader is missing, the log has. Flushing first means the containment runs the other way, so there can exist chunks in D that are not in W. Recovery, in either world, can only serve what is in W. In the first world that is enough by construction, because everything the reader lacks is there to be sent again, and the worst outcome is sending something twice. In the second, the reader's cursor can point at a position past the end of W, and the server is being asked for a chunk that no longer exists anywhere — not in the log, and not at the provider, whose connection died with the process that held it.

for await (const event of upstream) {
  const seq = ++cursor;
  await log.append(generationId, seq, event.kind, event.data); // durable
  subscriber?.write(sse(seq, event.data));                     // best effort
}
await log.finalize(generationId, event.finishReason, event.usage);

The await on the append and the absence of one on the write is the whole design in two lines. The durable write is on the critical path and its failure aborts the generation; the client write is a side effect whose failure is uninteresting, because the previous chapter established that the client's presence is not what the generation depends on.

What a per-token write costs, and how to avoid paying it#

Providers emit one delta event per token, so a 900-token answer is about 900 events and a 6,200-token report is about 6,200. Writing each one as its own row is where a naive implementation gets expensive. A single-row insert into Postgres in the same availability zone costs roughly 1.2 ms of round trip, and awaiting each one serialises the relay: 6,200 × 1.2 ms is 7.4 seconds added to an 88-second generation, which is survivable, but the aggregate is not. Two hundred concurrent generations decoding at 45 tokens per second is 9,000 inserts per second against a database that also serves your application.

Two multiplications carry that argument and they are worth separating, because they answer different questions. The first is per generation: 6,200 sequential round trips at 1.2 ms each is a little over seven seconds, and those seconds add to the 88 rather than overlapping with them, because the await holds that generation's loop until the row is acknowledged. The second is per cluster and does not mention the answer length at all. Two hundred generations each emitting 45 tokens a second emit nine thousand tokens a second between them, and one row per token makes that nine thousand inserts a second regardless of whether the answers are short or long. A cost you can absorb per generation becomes a cost you cannot absorb per fleet, without a line of the code changing between the two readings, which is the general shape of every capacity surprise in this topic.

Coalescing fixes it. Accumulate deltas and emit a chunk on whichever comes first: a token count, or a time bound that keeps the stream feeling live. Eight tokens or 80 milliseconds is a reasonable pair, because 80 ms is below the threshold at which text stops looking like it is being typed, and eight tokens at 45 tokens per second is 178 ms of decode, so the time bound is usually the one that fires. That takes 6,200 events down to about 500 rows and 9,000 writes per second down to roughly 1,100.

Which of the two bounds actually fires is a property of the decode rate rather than of the configuration, and it moves during a single answer. The chunk closes on whichever arrives first: the eighth token, or the eightieth millisecond. A fast stretch of decoding reaches eight tokens inside the window, so the count binds and the chunk carries eight. A slow stretch — a reasoning pause, a tool call being executed, a queue wait upstream — lets the timer fire with two or three tokens in hand, so the timer binds and the chunk carries fewer. The row count is therefore the token count divided by however many tokens the average chunk turned out to hold, and that average is not something the code sets. What the code does set is the ceiling: no chunk holds more than eight, so the write rate can never rise above the token rate divided by eight however fast the provider decodes.

The coalescing window is also the size of your loss window. Anything buffered and not yet written dies with the process, so an 80 ms window means a pod kill costs at most 80 ms of tokens — three or four tokens, which the replay in the next chapter will smooth over. A 2-second window would be cheaper and would lose a sentence.

Redis Streams are the common choice and the reference design behind the AI SDK's Chatbot Resume Streams guide, because XADD plus XREAD BLOCK gives you append and live tail with one primitive, and the durability you actually need is "survives the pod", not "survives the database". Object storage is the wrong shape for chunks and the right shape for the finished answer.

That distinction matters more than the store choice. The chunk log is scaffolding with a lifetime measured in minutes: it exists so that a client which dropped can catch up, and once the generation is terminal and the assembled answer has been written to wherever answers live, the log is redundant. Give it a TTL — a time to live, after which the store deletes the entries itself — of an hour, and trim it. Systems that keep the chunk log forever end up with two representations of every answer and eventually a bug where they disagree.

Ordering has to come from one writer#

Sequence numbers are only meaningful if exactly one thing assigns them. The owning task increments a local counter as it reads the upstream stream, which is safe because there is one such task per generation by construction. Substituting storage-side ordering — a Redis stream id, an auto-increment column, an insertion timestamp — works right up until a retry causes two writers to exist briefly, at which point the log has an order that does not match the order the tokens were sampled in, and the answer reads as scrambled text with no error anywhere.

The retry is the step that makes a borrowed order fail, so it is worth following once. A write that times out has not necessarily failed; it may have committed and lost its acknowledgement on the way back. The owning task retries, and for a moment two writes for the same chunk exist in the store. Under a writer-assigned sequence both carry the same number, so the second collides and is discarded. Under a storage-assigned order the second is given whichever position happens to be free when it lands, which is behind the chunks written while the retry was in flight — so an early chunk ends up ordered after later ones. Nothing errors, because every row is well-formed and the resulting order is a perfectly valid order. It is simply not the order the tokens were sampled in, and a fold over it produces prose with two clauses transposed.

The same reasoning applies to concurrency within one generation. If you parallelise the write, so that chunk 30 is inserted while chunk 29 is still in flight, the log can be missing an interior sequence number at the instant a reader queries it. A reader that treats "the highest seq I can see" as the end of the log will stop early. Either keep the append serial, or have readers stop at the first gap rather than at the highest number, and be explicit about which rule you chose.

Where the log stops short#

It does not make delivery correct. A log tells you what was produced and in what order; it says nothing about what the reader actually rendered, and a server that assumes the two are the same will either replay text already on screen or skip text that never arrived. That gap is the subject of the next chapter, and it is the reason this one stops at "the answer survives" rather than claiming "the client resumes correctly".

Resumption needs a second number that this chapter never produces. The log's last entry records what was produced; what a returning reader needs is what arrived, and the write-before-flush ordering guarantees on purpose that those two are different. A server that treats its own last_seq as the reader's frontier is asserting the containment runs the wrong way, which is exactly the assertion the ordering above was chosen to avoid making.

It is also not chat history. The two get conflated because both store model output, but they have different granularities, different retention, and different consumers: history stores one assembled message per turn and is read by the prompt builder, while the chunk log stores hundreds of fragments per message and is read by a reconnecting browser within the next few minutes. Merging them produces a table with a hot append path and a hot read path and a retention policy that cannot satisfy either.

And it does not rescue you from a provider failure mid-stream. If the upstream connection breaks at token 3,000 of a 6,200-token report, you have a durable prefix and no way to continue it — a new request with the same prompt produces different text, and feeding the prefix back as an assistant turn to be continued works only on models and APIs that support prefill, and even then produces a seam. Persisting the prefix is still worth doing so you can show the user what was produced and mark the generation failed rather than leaving it ambiguous, but the honest position is that a mid-stream provider failure costs you the generation.

What the log does give you is a generation that outlives every process involved in producing it, at the price of one durable write per coalesced chunk and a retention policy. The two rules that carry the design are short enough to hold in review: append before you flush, because duplication is repairable and loss is not; and assign sequence numbers from the one task that owns the generation, because an order borrowed from the storage layer is an order that can be wrong under retry.

With the answer durable, the remaining question is where a returning reader should be sent. The log knows what was produced. It does not know what was seen.