Concurrent turns and lost updates

Two requests arriving on one conversation at nearly the same moment is a read-modify-write race, the shape every backend engineer has already met as balance = read(); write(balance - 50), and the failure it produces is a plain lost update: two writers compute from the same snapshot, and one of their effects vanishes. What makes it worth a chapter of its own is the width of the window. In the banking version the gap between read and write is microseconds, so the race is rare enough that people ship the broken version and get away with it for years. Here the gap contains a model call, and a model call takes seconds. When the p50 is 4 seconds and the p99 is 20, the "race" is not a race at all — it is the normal outcome of a person typing a follow-up before the first answer arrives.

"Inside each other's window" is worth making literal, because the width of that window is the whole of the reason this needs a chapter. The window opens when a request reads the conversation and closes when it writes back, so its width is whatever the model call takes. A second message typed while the first answer is still streaming falls inside it, and so does a retry fired by a client that gave up waiting, and so does a webhook that happens to land. Whether a collision occurs is therefore not a question about luck but a question about the ratio between that width and the gap between turns, and at the median the width is four seconds — less time than it takes to read an answer and decide it was not the one you wanted.

The damage from one collision is in two places and only one of them is obvious. The obvious half is the append: two rows computed for the same position, one of which is rejected or silently wins. The half that costs more is the read, because each request assembled its outbound array from a snapshot that did not contain the other's turn, so a reply was generated against a conversation that no longer exists and is now stored as the most recent thing the assistant said. Everything after it inherits that.

Walking the collision#

Both requests execute the loop from chapter 1 — insert the user row at the next position, read the conversation, call the model, insert the assistant row — faithfully. Neither does anything wrong on its own.

What allows them to overlap is not a defect in either request but three ordinary properties of the path they take. An HTTP server handles requests concurrently by design, so two calls on one conversation are two independent executions of the same handler with nothing between them. Each execution borrows its own database session, and a session's statements are isolated from another session's uncommitted work rather than ordered against it — isolation means "you cannot see my half-finished writes", not "you must wait for me". And the handler's own steps are separate statements rather than one indivisible operation, so there is no interval during which the conversation belongs to one request and no other request may touch it. Concurrency here is the default behaviour of every layer involved, and serialising turns is the thing that would have to be added.

Tab A computed the position for its assistant row before it made the model call, as next_seq + 1, which was 11 at the time and is now occupied by tab B's user message. What happens next depends entirely on a schema decision made months earlier. With the UNIQUE (conversation_id, seq) constraint from chapter 1, the insert raises SQLSTATE 23505, which is Postgres's unique-violation error, and whether the customer's turn survives depends on how that exception is handled — a bare except Exception: log.warning(...) in the write path turns it into a silently dropped assistant message, and a 500 returned after the model has already been charged for and streamed a reply is not much better. Without the constraint, both rows sit at position 11, ORDER BY seq has a tie, and the planner returns them in whichever order the scan produced, which can differ between two reads of the same table.

The repair that suggests itself first — take a lock before computing the position — does not fit the shape of the problem, and the reason is worth stating because it recurs. A database lock is held on something that exists: a row, a page, a named advisory token. What tab A needed to reserve was the absence of a row at position 11, and an absence is not something the storage engine has a handle for. SELECT max(seq) reads the rows that are there and returns a number about the row that is not, and locking the rows it read protects nothing, because the row that invalidates the number has not been written yet. Every workable answer in the next chapter therefore locks something other than the missing row.

The timing of the constraint is worth pinning down as well, because a unique index reads as something that prevents the collision when what it does is notice one. It makes the bad state impossible to store; it does nothing to make the bad state impossible to compute. The index entry for a position is created when a row is inserted, which is after both requests have read, after both have assembled an outbound array, and after at least one model call has been paid for and streamed to a customer. A second insert on an occupied position does not fail immediately either: it waits on the transaction that got there first, and raises SQLSTATE 23505 only once that transaction commits. Had the first transaction rolled back instead, the waiting insert would have succeeded. The constraint therefore reports the collision at the last possible moment, to the request least able to do anything cheap about it.

The lost or mispositioned row is only half of it. The more consequential damage happened earlier, at the two SELECT history steps. Tab B's model call did include the refund question, since B read after A's insert, but tab A's did not include the cancellation. So the refund answer was generated against a conversation where the customer had not yet changed their mind, and it is now stored as the most recent assistant turn. Every subsequent turn of this conversation inherits that: the model reads a history in which the last thing the assistant said was that a refund is available, and it will act consistently with that.

That row does not decay. History assembly rebuilds the outbound array from the stored rows on every call — the property from chapter 1 that made the design checkable in the first place — and the same property is what makes this damage permanent. A wrong row is not a stale view of the conversation that some later read will correct; it is the conversation. Nothing later in the loop revises it, because the model is handed the array and asked to continue it, and nothing in that arrangement marks one row as mistaken. A customer can contradict the assistant in a subsequent turn, but the contradiction lands as one more row underneath the wrong one rather than as a replacement for it.

Two tabs is only the most legible source of concurrent turns, and by the time a product has any real traffic it is not the most common one. A mobile client that loses its connection mid-request and retries produces two requests for one turn, seconds apart, and the first one is usually still running on the server. A tool-calling loop that posts intermediate results back into the conversation writes turns from a worker while the user is typing into the same conversation from their browser. An integration that ingests emails or webhook events into a support thread appends turns on a schedule that has no relationship to what the human is doing. All of these arrive as two writers on one conversation with a multi-second window between read and write, which is the same shape as the two-tab case and is not something a user interface can prevent.

The third failure mode in this family reaches other people's conversations, and it comes from a delivery path rather than from the table. Streaming responses need somewhere to publish to, and a common shortcut is a channel or cache slot keyed by conversation id — pending_reply:{conversation_id} in Redis, or a pub/sub topic of the same name. Two in-flight turns on one conversation write to the same slot, so tab A, polling or subscribed, receives the text generated for tab B. If the key is scoped more loosely still, which happens when a worker keys by user id so that a single reconnecting mobile client can find its stream, a reply generated in one conversation is delivered into a different one, and the user sees an answer about delivery slots appear in their returns chat. The stored rows in that case can be entirely correct while the user's transcript is not.

The reasoning that produces that key is worth following, because it is sound reasoning and it produces the bug anyway. A reply has to be published somewhere the reader can find it, and the reader is a browser tab that may have reconnected to a different server than the one generating the text. The only identifier both sides hold before the reply exists is the conversation id, so the key gets picked from what is available rather than from what is unique. The mismatch is that the key names a container while the thing being stored is one item out of it, and a container holds more than one in-flight item precisely when the user is doing what this chapter is about. Keying by user id widens the same defect by one level, so the container becomes every conversation that user has open.

None of this is hard to understand once it is written down, and all of it is hard to find, for one reason worth naming plainly. At the moment of failure every signal the system emits is a success signal: two requests were accepted, two model calls completed, two replies were rendered, and every row that was written is a well-formed row. The damage is a relationship between rows rather than a property of any one of them, and nothing emits a metric about a relationship until somebody has added the constraint or the query that turns one into a number. Anybody debugging this from an off-the-shelf dashboard is looking at a dashboard that is telling the truth.

Reproduce it before you fix it#

A concurrency bug that has never been observed on demand tends to get "fixed" by a change that makes it rarer, which is indistinguishable from a fix until the traffic doubles. Making it deterministic takes a barrier and a slow fake model.

Both halves earn their place. The barrier is the start event below: two coroutines are created, each waits on that one event before doing anything, and setting it releases both at the same instant. Without a barrier the second task begins whenever the first happens to yield, so which statement interleaves with which is a property of the scheduler and varies between runs. The slow fake model does the other half by holding the window open on purpose. A real call is four to twenty seconds and an instant fake is none at all, so a suite built on an instant fake has been testing a version of the system whose window was closed for it by the test double.

import asyncio, pytest

class SlowFakeModel:
    """Widens the window to the p99 of a real call, deterministically."""
    def __init__(self, delay=2.0):
        self.delay = delay
        self.calls = []

    async def create(self, *, model, messages):
        self.calls.append([m["content"] for m in messages])
        await asyncio.sleep(self.delay)
        return reply_for(messages)

@pytest.mark.asyncio
async def test_two_tabs_one_conversation(db, monkeypatch):
    convo = await seed_conversation(db, turns=9)
    model = SlowFakeModel()
    monkeypatch.setattr(app, "client", model)

    start = asyncio.Event()

    async def send(text):
        await start.wait()                       # release both at once
        return await handle_turn(db, convo, text)

    task_a = asyncio.create_task(send("Can I get a refund for order 88231?"))
    task_b = asyncio.create_task(send("Actually cancel the whole order instead."))
    start.set()
    await asyncio.gather(task_a, task_b, return_exceptions=True)

    rows = await db.fetch(
        "SELECT seq, role, content, in_reply_to FROM messages"
        " WHERE conversation_id = $1 ORDER BY seq, id",
        convo,
    )
    assert [r["seq"] for r in rows] == list(range(1, 14))          # contiguous, no gaps or dupes
    assert sum(r["role"] == "user" for r in rows[9:]) == 2         # both questions survived
    assert sum(r["role"] == "assistant" for r in rows[9:]) == 2    # both answers survived
    for r in rows:
        if r["role"] == "assistant":
            question = next(x for x in rows if x["seq"] == r["in_reply_to"])
            assert question["role"] == "user"
            assert answers(r["content"], question["content"])

The assertions are arithmetic on the fixture, and following it once is what makes them readable. The fixture leaves the conversation at position nine, which is where the sequence diagram's SELECT max(seq) picks it up. Two concurrent turns contribute two user rows and two assistant rows, so a correct run ends with four more rows than it began with, and nine plus four is thirteen. The expected list is therefore the positions one through thirteen with no gaps and no repeats, which is what range(1, 14) spells: Python's range excludes its upper bound, so it stops one past the last position it is meant to include. rows[9:] then skips the seeded rows, which is why the two counts underneath it are statements about the four new rows alone — two of them user, two of them assistant.

Two details in that test matter more than the mechanics. The first is in_reply_to, a column holding the seq of the user message a given assistant row responds to, written from the value the handler captured before it made the call. Without it, "the reply is paired with the wrong question" is something a human has to notice by reading a transcript; with it, the pairing is data and an assertion can check it. It costs one integer column and it is the difference between a test that catches this class of bug and a test that catches only the crude version where a row disappears entirely.

Capturing the value before the call, rather than working it out from the table afterwards, is what makes the column independent evidence rather than a second copy of the same guess. The pairing then records what the request believed at the moment it built its array, which is a fact no later write can alter. Every other signal an audit could use — adjacency in the transcript, ordering by seq, ordering by created_at — is derived from exactly the thing the race corrupts, so it agrees with the corruption instead of exposing it. A check built on those signals reports a tidy conversation. A check built on in_reply_to reports an assistant row whose question is not the one above it.

The second is what the test refuses to assert on. Both HTTP responses in the failing run are 200, both carry fluent text, and both look right to the caller who received them — tab A got a refund answer to a refund question. The corruption is in what was persisted and in what the next turn will therefore assemble, so asserting on the response body proves nothing. Ordering, contiguity, ownership by conversation, and reply pairing are the properties under test, and they are all read back out of the table after the fact.

Running the same test with delay=0.0 is a useful control, because it usually passes. The window closes, the two requests serialise by accident, and a suite that only ever exercised the fast fake would have shipped this. Any fake model used in tests that touch concurrency should have a delay that is at least as long as the real p50.

The gap between a test that usually catches a bug and one that always catches it is not a difference of degree. A run that fails some unpredictable fraction of the time is read by whoever sees it as flakiness, and the standard response to flakiness is a retry, so a probabilistic concurrency test turns a real defect into a nuisance that eventually gets suppressed. The delay does not create the race — the race is in the handler either way — it removes the scheduler's discretion over whether this particular run will expose it. That is the property to demand of any test in this family, and the way to establish it is to run the test against the unfixed code repeatedly and confirm that it never passes.

Production needs the same properties expressed as monitoring, because a test proves the fix works on the machine that ran it and says nothing about the conversation a customer opened in three places. Three cheap signals cover most of it. Count unique-violation errors on the messages table by conversation, since a rate that is not zero means requests are still computing positions from stale snapshots even if the handler recovers gracefully. Run a periodic audit query that looks for conversations whose seq values are not a contiguous run from 1, or that contain two assistant rows with the same in_reply_to, and alert on the count rather than on individual rows. And record, per turn, how many messages existed at read time against how many exist at write time; a non-zero difference is precisely the condition the next chapter's policies exist to handle, and knowing how often it occurs in your traffic is what makes the choice between them an informed one rather than a preference.

-- conversations where positions collided or a reply lost its question
SELECT conversation_id, count(*) AS suspicious_rows
FROM (
  SELECT conversation_id, seq, count(*) OVER (PARTITION BY conversation_id, seq) AS dup
  FROM messages
  WHERE created_at > now() - interval '1 day'
) t
WHERE dup > 1
GROUP BY conversation_id;

That query returns the affected conversations rather than a total, and its inner line is where the work happens. count(*) OVER (PARTITION BY conversation_id, seq) computes, for every row, how many rows share both its conversation and its position, and attaches that number to the row without collapsing anything — which is why it is a window function and not a GROUP BY. A GROUP BY would hand back the positions that are duplicated and discard the rows themselves, and the rows are what somebody on call needs in order to read the transcript and see which reply went to which question. The outer WHERE dup > 1 keeps only rows that have a twin, and the final grouping reduces those to one line per conversation. Read the comment above it against what it actually does, though: it names two conditions, and the query implements the first. Positions that collided are duplicates and are caught here, while a reply that lost its question has a position all of its own and is invisible to this query, so that half needs its own pass over in_reply_to.

Detection without a decision#

The unique constraint is not a solution. It converts silent corruption into a loud failure, which is a real improvement in diagnosability and no improvement at all for the customer whose cancellation vanished with a 500. Something still has to decide what the losing request does.

Raising the isolation level does not decide that either. Running the read-and- insert under SERIALIZABLE will make Postgres detect the dangerous interleaving and abort one transaction with a serialisation failure, which is again a correct detection and again leaves you holding a request that must now do something. It is also of limited use here because the expensive, slow part of the turn — the model call — sits outside any transaction you would be willing to hold open; a database transaction spanning a 20-second network call to a third party is a different production incident.

That detection is also narrower than the sentence above makes it sound, and the narrowing matters before anyone proposes the setting as a fix. SERIALIZABLE aborts one of two transactions when the read one of them made and the write the other performed form the dependency structure it looks for, which requires that the read and the write derived from it sit inside the same transaction on each side. The handler in chapter 1 does not have that shape: the position is read and the user row written inside one transaction, the history is read outside any transaction at all, and the assistant row is written inside a third. Raising the isolation level on the first of those leaves the other two exactly as exposed as they were. The step to make somebody demonstrate is not that the setting is enabled, but that a barrier test against the real handler now fails one of the two turns.

Widening the transaction to cover the whole window is the next thing suggested in review, and it fails for a reason worth being concrete about. The window that needs protecting runs from the history read to the assistant insert, and the model call sits in the middle of it. Holding a transaction open across that call pins a pooled connection and an open snapshot for as long as the provider takes to answer, so the size of the connection pool stops being a function of how much database work there is and becomes a function of how many model calls are in flight. The pool is then exhausted by requests that are, at that moment, doing nothing to the database at all. The window is wide because a third party is sitting inside it, and no database setting reaches into a third party.

This is also a different problem from the stale in-process cache of chapter 1, and worth keeping separate because the two get conflated under "concurrency issues". That one was a read-path problem, where the underlying rows were correct and one process was looking at an old copy of them; fixing it required removing a cache. This one is a write-path problem, where the rows themselves end up wrong and every process reading them afterwards, including a freshly restarted one, sees the same corrupted conversation. No cache invalidation helps.

Nor is it fixed by treating it as a client-side problem. Disabling the input box while a turn is in flight makes the two-tab case slightly less common and does nothing about a mobile client that reconnects and retries, a user with the same conversation open on their phone and laptop, or an integration that posts turns from a webhook. Any of those produce two concurrent writes against one conversation, and the server is the only place that can arbitrate.

That last claim is a statement about who can see what, and it is worth checking rather than accepting. The two clients cannot see each other, since a browser tab has no way of learning that another tab is mid-request on the same conversation. The model provider cannot see it either: the endpoint retains nothing between calls and associates no two calls with one another, so what reaches it is two unrelated requests that happen to share most of their text. The database sees both, because both eventually write to it, and the handler in front of the database is the only code that runs on both paths. Arbitration has to live where both requests are visible, and that is the only place they are.

Concurrent turns on one conversation are ordinary lost updates made routine by a model call sitting in the middle of the window, and the reason to treat them as a first-class case rather than an edge case is that the corruption is durable. A lost banking update costs one transaction; a lost turn is baked into the array every later turn assembles, so a customer sees an assistant behaving strangely days after the collision that caused it. The work that makes this tractable is mechanical — one in_reply_to column, a fake model with a realistic delay, a barrier in a test, and an audit query on contiguity — and it turns "we think there might be a race" into a failing assertion.

What is missing is a policy: an explicit statement of what happens to the request that arrives second. There are three defensible ones — reject and retry against fresh history, serialise turns behind a per-conversation lock, or let the conversation branch and reconcile afterwards — and they differ in what they cost, in how they fail, and in what the losing tab has to show a person who is sitting there waiting. The next chapter takes them one at a time.