Sequencing policies for concurrent turns

Chapter 4 established that two turns landing on one conversation within a few seconds each read the same stored history and then collide on the write, and that a unique constraint on (conversation_id, seq) turns that collision into a detected error rather than silent corruption. Detection is not a policy. The request that lost still exists, a person is still watching a spinner in the tab that sent it, and a model call has probably already been paid for on its behalf.

One property of that race decides which policies are even available, which is that the gap between the read and the write is a model call rather than a few instructions. The obvious way to close a gap between a read and a write is to hold a database transaction across it, and chapter 4 rules that out: a transaction spanning a twenty-second call to a third party pins a connection and holds a row lock for the whole of it. That single exclusion leaves exactly three families of answer — let the gap stay open and check on the way out whether anything changed, take a claim that lives somewhere other than the database transaction, or stop insisting the history is a single line at all.

This chapter compares the three mechanisms that decide the loser's fate. Optimistic concurrency puts a version number on the conversation and checks it at write time, rejecting a write built on a stale read. Pessimistic locking takes an exclusive claim on the conversation before reading and holds it until the append is done. Branching declines to serialise at all and lets the conversation become a tree. They differ in mechanism, in what they waste, and — the part that decides most real arguments — in what the losing tab is able to show.

Three ways to decide who loses#

Optimistic concurrency puts a version on the conversation and checks it at the moment of writing rather than at the moment of reading. That is not a convenience. At the moment of the read there is nothing to check, because the history returned is the history as it stands; what makes it wrong is an event that has not happened yet, and no test performed at read time can detect that. The earliest instant at which "is this still the current history?" has an answer is the instant the losing request tries to become part of it.

ALTER TABLE conversations ADD COLUMN version integer NOT NULL DEFAULT 0;

async def handle_turn(conn, conversation_id, user_text, attempt=0):
    history, version = await load_history_and_version(conn, conversation_id)
    messages = [SYSTEM_PROMPT] + history + [{"role": "user", "content": user_text}]
    reply = await call_model(messages)                      # 4-20 s, no lock held

    async with conn.transaction():
        bumped = await conn.execute(
            "UPDATE conversations SET version = version + 1"
            " WHERE id = $1 AND version = $2",
            conversation_id, version,
        )
        if bumped == "UPDATE 0":                            # someone else appended meanwhile
            raise ConversationChanged(version)
        await append_pair(conn, conversation_id, user_text, reply, base_version=version)
    return reply

The UPDATE ... WHERE version = $2 is a compare-and-swap, and the zero-rows case is the whole policy: it means the history this reply was generated against is no longer the current history. The losing request has already spent a model call — 3,000 input tokens and 400 output tokens is about $0.0007 at $0.15 and $0.60 per million — and it must throw that reply away, reload, and generate again against a history that now includes the other turn. Money is not what it costs; the second call costs the user another 4 to 20 seconds of waiting, and it costs you the discipline of bounding retries so that a pathological conversation cannot spin.

Calling that statement a compare-and-swap is a claim that the comparison and the increment cannot come apart, and that property is supplied by the database rather than by the code around it. UPDATE conversations SET version = version + 1 WHERE id = $1 AND version = $2 is one statement, and Postgres takes a row lock on the matching row before it applies the change. A second request running the same statement against the same row cannot evaluate its WHERE clause in the middle of the first one: it blocks on that row lock, waits for the first transaction to commit, and only then re-reads the row. The value it finds has already been incremented, so its own version = $2 predicate matches nothing and the statement reports zero rows changed. The conflict is a predicate that was true when the request read it and false by the time the request got to use it.

The waste is a model call, and the sum is worth doing on the figures already quoted rather than taking $0.0007 on trust. A price of $0.15 per million input tokens scales to any smaller count by proportion, because every token is billed at the same rate — so a thousand input tokens cost a thousandth of it, which is fifteen hundred-thousandths of a dollar, and three thousand cost three times that, or forty-five hundred-thousandths. Output is metered separately at its own rate: $0.60 per million makes a hundred output tokens six hundred-thousandths of a dollar, and four hundred come to twenty-four hundred-thousandths. The meters are independent, so the total is their sum — sixty-nine hundred-thousandths of a dollar, which is the $0.0007 above once rounded to the same places. A fraction of a tenth of a cent is what "money is not what it costs" amounts to when the figures are actually multiplied out, and it is why the seconds are the term that matters.

One step in the retry that follows is easy to assume and worth checking, because assuming it wrongly is how a retry turns into a duplicate. The losing request generated a reply, so the natural reading is that something must have been written before the conflict was noticed. Nothing was. The compare-and-swap and the append sit inside the same conn.transaction() block, and the handler raises ConversationChanged before reaching append_pair, so the transaction unwinds with no rows added. The retry is not repairing a partial write; it is starting from a conversation that never heard from this request at all.

That safety holds under a condition worth making somebody demonstrate rather than assert. The handler above returns reply only after the transaction commits, so a discarded generation is discarded before anyone sees it. A handler that streams tokens to the browser as they arrive has already shown the user a reply it is about to throw away, and no transactional care on the server can retract text that is on screen. The two are compatible only if the stream is held until the version check passes, or the interface can replace a reply in place.

Pessimistic locking takes the lock before reading and holds it until the append is done. Doing it correctly means a lease — an expiry attached to the lock so that a pod which dies mid-turn does not freeze the conversation permanently — with renewal rather than a fixed timeout, because the only honest answer to "how long will this take" is "longer than you want to commit to".

async def with_conversation_lock(rdb, conversation_id, fn, lease_ms=15000):
    token = uuid4().hex
    key = f"conv:{conversation_id}"
    if not await rdb.set(key, token, nx=True, px=lease_ms):
        raise ConversationBusy()

    async def heartbeat():
        while True:
            await asyncio.sleep(lease_ms / 3000)
            await rdb.eval(EXTEND_IF_OWNER, 1, key, token, lease_ms)  # renew only if still ours

    hb = asyncio.create_task(heartbeat())
    try:
        return await fn()
    finally:
        hb.cancel()
        await rdb.eval(RELEASE_IF_OWNER, 1, key, token)   # never DEL blindly

What makes rdb.set(key, token, nx=True, px=lease_ms) a lock rather than a hopeful convention is a property of the server it is sent to. Taking a lock is really two operations — find out whether anyone holds the key, and claim it if nobody does — and a lock is exactly what you do not have if another request can slip between them. NX fuses the pair into one command, and Redis executes commands one at a time, so nothing runs part-way through it: of two requests arriving together, the first gets a key that did not exist and a true return, the second gets a key that now does and a false one. PX attaches the expiry inside that same command, because a claim created without one outlives any pod capable of releasing it.

Both Lua scripts check the token before acting, because a lock released by whoever happens to call DEL is a lock that one request can drop on another's behalf. The heartbeat converts "lease longer than p99" into "lease longer than one heartbeat interval", so a dead pod frees the conversation in about five seconds instead of thirty. It does not remove the failure mode; a pod that is alive but stalled on a hung HTTP connection keeps renewing a lock around a call that will never return, so the lease still needs a hard ceiling and the handler still needs a client-side timeout that is shorter than it.

"One request can drop another's lock" is a sequence rather than a slogan, and the ending is not the one people expect. Request A takes the lock and overruns its lease, so the key expires while A is still generating. Request B finds no key, takes the lock legitimately, and starts work. Request A finishes and runs its release; a blind DEL there deletes the key B is holding, because A's own key is long gone. Request C then finds the conversation free and enters it while B is still inside. The damage is not that A released late — it is that A's release admitted C to a region B was supposed to have alone, so one overrun propagates into a violation between two requests that never overran anything. Comparing the token stops it: A's release finds a value that is not A's and does nothing.

The comparison and the deletion also have to happen with nothing in between, which is the NX requirement again at the other end of the lifecycle. A GET followed by a DEL is two round trips with a gap, and the lease can lapse and be re-acquired inside it, at which point the DEL removes a lock the GET never saw. One Lua script closes that gap because Redis will not interleave another client's command with a script already running.

"About five seconds instead of thirty" is the heartbeat's doing, and both figures come out of the sleep interval, which is why that interval is written as a division rather than a constant. lease_ms / 3000 folds two conversions together: a division by three, so renewal runs several times per lease and one missed beat is not fatal, and a division by a thousand, because the lease is quoted in milliseconds and asyncio.sleep takes seconds — on the fifteen-second lease in the signature, a beat every five seconds. Without renewal the lease must exceed the slowest call you will tolerate, which is where thirty comes from, and a pod dying while holding it freezes the conversation for that whole period. With renewal the lease need only outlive one beat, so a dead pod stops renewing and the conversation frees itself in about five. Cancelling the heartbeat before releasing, on the line above, stops a renewal landing after the release and re-creating a lock no live process knows the token for.

The Stolen transition is the one that decides most arguments about locks. A lease held for less time than the operation it protects is, for the duration of the overrun, the same thing as no lock at all — except that the second request has now also paid for a generation. Raising the lease to cover the tail makes the second tab of an ordinary user wait that long before its turn even starts, and leaves a conversation frozen for the same period after a pod dies mid-generation. There is no lease length that is both longer than the tail of the model call and short enough to be an acceptable wait, because those are the same number.

That last sentence is short enough to slide past, and it collapses three constraints on one number. The lease has to be longer than the operation it protects, or it lapses mid-operation and stops protecting anything. The waiting request is blocked or refused for as long as the holder can legitimately hold, so the lease is also the worst wait a second tab can be made to sit through. And a pod that dies holding the lease leaves the conversation unusable until it expires, which the user experiences as an assistant that has stopped responding, so the lease is additionally the outage a crash produces. Nothing about the mechanism lets those three be tuned apart.

Concluding from that a lock is the wrong tool is worth the working, because reaching for one is not a careless instinct. Mutual exclusion is the standard answer to a read-modify-write race and stays the correct one when the critical section is short, local and bounded. What changes here is what the critical section contains: a network call to a third party whose duration is a distribution with a long tail rather than a value, against a lease that is a prediction of that duration made before the call starts. The lock did not remove the race. It converted it into a wait governed by a timer that knows nothing about the operation it is timing, and the race returns in full whenever the timer guesses low.

Branching declines to serialise at all. Each turn records the message it extends, so two concurrent turns produce two children of the same parent and both succeed, and the conversation becomes a tree that the read path must resolve into a line. This is the mechanism behind edit-and-regenerate in consumer chat products, where forking is a feature the interface exposes deliberately. As a concurrency policy it removes the wasted call and the wait entirely, and relocates the whole problem to two places: assembly, which must now pick a branch, and the human, who must now be told there are two.

Both of those relocations follow from one change, and it is a change in what a write claims rather than any new coordination. A seq is a position in a line, and a line has one slot per position, so two requests that computed the same next position are competing for something only one can have — the collision the unique constraint from chapter 1 exists to catch. A parent pointer claims nothing. It names a row that already exists and is not being modified, and any number of rows may name the same parent without contradicting each other. The contention was not resolved; it stopped being contention, because the writes no longer want the same thing.

Picking a branch is a bigger job than it sounds, because the read path changes shape rather than gaining a clause. Assembling a line meant reading a conversation's rows in position order, which a database does in one pass. Assembling a tree means starting at a chosen leaf, following parent pointers back to the root and reversing what you collected — and, before any of that, deciding which leaf. Nothing in the stored rows answers that, because two siblings are equally valid continuations by construction, so the choice comes from outside the data. That is the step most often skipped when branching is proposed as a concurrency fix: the policy decided nothing, it deferred the decision to whoever reads the conversation next, and whether that is a feature depends entirely on who that reader is.

The same three, side by side#

For a customer support tool where a model call takes 4 to 20 seconds, optimistic concurrency is the one to ship, and the reason is the conflict rate. Humans in a support conversation send a turn every 10 to 60 seconds, so two turns overlapping is an edge case measured in low single-digit percentages — the two-tab user, the impatient follow-up, the mobile client that retried. Paying an extra call on 1% of turns, and paying nothing on the other 99%, beats paying for lock infrastructure on 100% of them, and the failure mode when the mechanism misbehaves is a request that gets rejected rather than a conversation that gets corrupted.

That conflict rate carries the argument, and the reason it is low is not the one the figures suggest. The collision window is the model call, 4 to 20 seconds wide, against turns arriving every 10 to 60 seconds — divide one by the other and overlap looks common. What keeps it rare is that a single person is self-serialising: they send a message, wait for the reply, and only then decide what to say next, so their second turn cannot start until their first has finished. Overlap needs two independent senders on one conversation, which is what the two-tab user, the impatient follow-up and the retrying mobile client have in common. Conflicts are rare because the window is almost never entered twice, not because it is narrow.

That makes the rate a property of a particular product rather than of chat, and the conditions are worth stating. It stays in low single digits only while every client waits for a reply before allowing another turn. An interface that leaves the composer live during generation, a client that retries a timed-out request without a key letting the server recognise it as the same turn, or an automated caller driving the conversation on a timer each break that assumption, and the last breaks it by design. What a review should ask for is the conflict rate measured on the traffic in question — how often the version moved between read and write — not the assurance that conflicts are rare in general.

The lock loses because it must be held across an operation whose duration you do not control and cannot bound, and every mitigation for that — longer lease, heartbeat, ceiling, client timeout — is a component that can fail on its own and whose failure reintroduces the interleaving it was bought to prevent. Branching loses on the audit requirement rather than on mechanics. A support conversation is going to be read by a human agent taking over from the assistant, and possibly by a regulator; "there were two versions of this conversation and the customer saw one of them" is not a transcript anybody can act on. Branching earns its place where the fork is meaningful to the user, as it is in a writing assistant where regenerating an answer is a deliberate act.

What the losing tab shows#

A policy is not finished when the server behaviour is decided, because the losing request corresponds to a person watching a spinner, and each policy dictates something different on screen.

With bounded silent retry, the first conflict shows nothing at all. The user's message stays in its pending state, the retry runs against fresh history, and the reply that arrives is coherent with the turn the other tab added — which is the correct outcome and worth the extra seconds. That silence has a budget: one retry, or two, and a visible message when it is exhausted. Retrying indefinitely produces a tab that spins for a minute against a conversation somebody else is actively driving, which is worse than telling the truth.

Leaving the message pending is honest rather than merely convenient, and the reason is the rollback from earlier in the chapter. A pending message claims the turn is not resolved yet, and after a failed compare-and-swap that claim is exactly true: the transaction wrote nothing, so there is no stored row the interface is contradicting. Staying quiet is truthful only while the server has genuinely left no trace — the same condition that made the retry safe, and the one a streaming handler gives up.

When the budget runs out, the honest thing to render is a conflict rather than an error. "This conversation was continued somewhere else — reload to see the latest" is accurate, actionable, and it is also what you should render when the user's message has already been answered elsewhere. Rendering 500 Internal Server Error for a condition you deliberately engineered trains users, and support, to treat a normal outcome as a fault.

It is not only people that get trained. A status code is the field infrastructure buckets on, so a server-error code for a conflict reports a product working exactly as designed as degraded — and a gateway configured to retry server errors will run the losing request again, outside the handler's own retry budget and without it knowing. A conflict has a code of its own for the same reason it has a message of its own: it states something about the resource, not about the server failing.

With a lock, the loser's experience is a wait with no explanation, and the interface has to decide at what point a wait becomes a statement. Fifteen seconds of unexplained spinner reads as broken; "another window is still replying" reads as a system that knows what it is doing. With branching, there is no way to keep the user out of it, since the whole design's premise is that both turns are real. The interface owes them a visible fork and a way to choose, which is a product surface with its own design cost, and it is why branching is rarely worth adopting purely to solve a concurrency problem.

None of the three is ordering in the semantic sense, and it is worth being clear about that limitation. When a customer asks for a refund and then says "actually, cancel the whole order", any of these policies will produce a well-formed transcript in which both turns are present and each has an answer, and the assistant will still have answered a question the customer had already withdrawn. Sequencing guarantees that the record is coherent and that the next turn sees everything; it does not give the model the ability to notice that turn 10 supersedes turn 9. That is a prompting problem, and it only becomes solvable because the array is correct.

Choosing among the three is mostly a question of who is sending the turns. Where they are human-paced and conflicts are a low single-digit percentage, optimistic concurrency puts the cost on the rare conflicting request and nothing on the rest, and its worst failure is a rejection you can render honestly. Where turns are machine-paced and generations are short, a lock is defensible because the mismatch between lease length and call duration stops being the dominant term. And branching is a product decision that happens to solve a concurrency problem, rather than a concurrency mechanism you should adopt for its own sake.

What all three policies share is a conversation that only grows. Every resolved turn appends two more messages, every assembly reads them all, and a support escalation that runs to sixty turns is sending an array that has been getting monotonically larger since turn one. There is a ceiling on that, it is a specific number for each model, and the next chapter is about finding out you are approaching it before the request does.