One duplicate is free, the other corrupts durable state

The support ticket says "the assistant answered me twice", with a screenshot showing two assistant bubbles containing near-identical text. The finance ticket, filed by a different team a week later, says one tenant's token usage is 6% higher than the sum of their conversation lengths. They are the same bug, and it starts in a config file on the mobile client:

val client = OkHttpClient.Builder()
    .readTimeout(30, TimeUnit.SECONDS)
    .retryOnConnectionFailure(true)
    .build()

Thirty seconds is a sensible default for a JSON API and a poor one for generation. On this service, p50 end-to-end was 4.1 s, p95 was 21 s and p99 was 34 s, so roughly 1.8% of requests took longer than the client was willing to wait. The client gave up, the socket closed, and OkHttp re-sent the POST. Nothing on the server noticed: the orchestrator had no idea the socket was gone, finished its generation at t=33s, committed the assistant turn and the usage receipt, and returned a response into a closed connection. The second attempt arrived at t=30.2s, read a history that did not yet contain the first answer, composed the same prompt, generated a second answer, and committed a second turn and a second receipt.

At 40,000 requests a day, 1.8% is 720 duplicated operations. The duplicated generation costs about $0.011 each at the service's average of 2,100 prompt and 500 completion tokens, so roughly $8 a day of wasted inference. That is the visible, expensive-looking part, and it is the part that does not matter.

The two duplicates are not the same object#

Two things were duplicated by that retry, and separating them is the whole design.

The first is the generation: a provider call that cost money, produced tokens, and then had its result committed. Its cost is bounded, denominated in dollars, and recoverable in the sense that spending $8 a day and stopping is a decision you can make on purpose. If the second attempt had generated an answer and then discovered the first had already succeeded, throwing that answer away would have left the system perfectly consistent and the user perfectly happy, minus $0.011.

The second is the commit: two rows in turns and two rows in usage_receipts. This one is not bounded and not recoverable, because it has changed what the system believes happened. The conversation history now contains two assistant turns with no intervening user turn, which means every subsequent request composes a prompt containing a duplicated answer, and the model's behaviour on that malformed history is nobody's specification. The invoice now contains a charge the customer will dispute, and the dispute lands on a human. Deleting the extra rows later requires knowing which pairs are duplicates, and after the fact there is no key that says so — same tenant, same conversation, timestamps 0.2 s apart, and legitimately-retried-by-a-user requests look identical. The duplicate that costs money is the cheap one, and the duplicate that writes is the one worth building a mechanism against.

The key has to reach the transaction that writes#

Idempotency here means one durable outcome per client-intended operation, so the key has to be something the client keeps stable across its own retries, and it has to be checked in the same transaction that performs the writes.

The client mints a UUID before its first attempt and re-sends it on retry:

POST /v1/chat HTTP/1.1
Idempotency-Key: 6f2b0f6e-6b1f-4a2f-9a01-53f3a03e1c5b
Content-Type: application/json

{"conversation_id":"c_8813","message":"summarise the March filings"}

The orchestrator claims that key before doing any work, and commits the outcome against the same row.

CREATE TABLE request_records (
  idempotency_key UUID PRIMARY KEY,
  tenant_id       TEXT        NOT NULL,
  request_hash    TEXT        NOT NULL,   -- hash of the semantic request body
  state           TEXT        NOT NULL,   -- 'in_progress' | 'committed' | 'failed'
  turn_id         UUID,
  response        JSONB,
  claimed_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
  completed_at    TIMESTAMPTZ
);

async def handle(ctx: RequestContext, body: ChatBody) -> Response:
    row = await db.claim(ctx.request_id, ctx.tenant_id, hash_body(body))
    if row is None:                                   # someone else holds the key
        existing = await db.get_record(ctx.request_id)
        if existing.state == "committed":
            return Response(200, existing.response)   # replay, no generation
        if existing.request_hash != hash_body(body):
            return Response(422, {"error": "idempotency_key_reuse"})
        return Response(409, {"error": "in_progress",
                              "retry_after_ms": 2000})

    result = await run_turn(ctx, body)                # the expensive part

    async with db.transaction():
        await db.insert_turn(result.turn)
        await db.insert_usage_receipt(ctx.request_id, result.usage)
        await db.insert_outbox(ctx.request_id, "call_completed", result.event)
        await db.complete_record(ctx.request_id, turn_id=result.turn.id,
                                 response=result.response)
    return Response(200, result.response)

db.claim is a single INSERT ... ON CONFLICT DO NOTHING RETURNING *, so the race between two simultaneous attempts is resolved by the primary key rather than by a read-then-write. The record moves through a small state machine, and the reason it needs three states rather than two is the in-flight case: the second attempt in the incident above arrived while the first was still generating, and a design that only records completions would have let it through.

Returning 409 with a retry_after_ms rather than blocking is the pragmatic choice for a mobile client, because holding the second connection open while the first generates doubles your concurrent connection count during exactly the latency spike that caused the retry. For a streaming endpoint there is a better option — attach the second connection to the first attempt's stream by request id, replaying from the buffer — and it is a meaningful amount of machinery for a case that a longer client timeout mostly removes.

The request_hash check exists because clients reuse keys by accident, usually by minting the key per screen rather than per request. Returning the first response to a different question is a much worse failure than rejecting it, so a mismatch is a 422.

Three patches that leave the shape intact#

The alternative proposal, and the one that usually arrives first in an incident review, is to make each layer idempotent: add a deduplication cache at the gateway, deduplicate turns at write time, and reconcile the usage records in a nightly job. Three changes, three teams, no argument, and it leaves the defect exactly where it was.

Gateway deduplication fails for the reason above — it records arrivals rather than outcomes, and it is per-instance unless you add a shared store. Write-time deduplication of turns needs a key that says two rows represent the same intended operation, and the only such key is the client's request id, which never reached the write because nobody plumbed it through; deduplicating on content hash instead breaks the moment a user legitimately asks the same question twice, which in a support context happens constantly. Nightly reconciliation of usage is the most revealing of the three, because it accepts that the system will produce wrong invoices and moves the correction into a batch job owned by someone who was not in the room. A reconciliation job also cannot distinguish a duplicate from a genuine repeat, so it either under-credits and generates disputes or over-credits and loses revenue, and either way somebody has to look at the output.

What the three patches have in common is that they add defences to layers without giving any layer the operation. The single change that fixes most of this is that the operation becomes a first-class object with a durable identity, claimed at the orchestrator before work starts and completed in the transaction that writes its effects. Once that object exists, the receipt is per-operation rather than per-provider-call, the turn is per-operation, and a second arrival with the same identity has an unambiguous answer available without any layer having to guess.

Why not the gateway, and why not the provider#

The gateway is the tempting place, because it already sees every request and is already shared, and putting deduplication there means no feature team has to think about it. It does not work for this failure, for a reason that has nothing to do with effort: the gateway cannot see the commit. Whatever it caches — an in-memory LRU, a least-recently-used cache that evicts whatever has gone longest untouched, or Redis with a 60-second TTL, a time to live after which the entry is dropped — records that a request with that key arrived, not that it succeeded. The first attempt was still generating when the second arrived, so a gateway that dedupes on arrival either rejects a legitimate retry of a request that later failed, or admits the retry and achieves nothing. Gateway caches are also per-instance unless you add the shared store, and once you have added the shared store you have built a weaker version of the request_records table one hop away from the transaction that would have made it correct.

Provider-level idempotency is a genuinely different mechanism aimed at a genuinely different problem. Most vendor SDKs send an idempotency header on their internal HTTP retries so that a network failure after the provider began work does not produce two billable inferences on their side. That protects your provider bill against transport retries below the client. It has no visibility into your turns table and no knowledge that the operation exists, so it prevents neither of the two duplicates discussed here.

The timeout that started it is also worth fixing#

Idempotency bounds the damage from a class of retry that will always exist, and it is not a reason to leave a 30-second timeout in front of a p99 of 34 seconds. Two changes remove most of the retries themselves. Raising the client read timeout above p99.9 — 45 seconds here — cuts the retry rate from 1.8% to under 0.1%, at the cost of a worse experience for the requests that genuinely hang, which is what the inter-token deadline from chapter 2 is for. Switching the endpoint to streaming is the stronger fix, because bytes arrive within about 1.4 seconds and the socket stays warm for the whole generation, so the HTTP client's read timeout never fires regardless of total duration. A service that streams and enforces idempotency has both the mechanism and the absence of the pressure that exercises it.

The request_records table needs an operational story of its own, since it grows one row per request forever if nobody decides otherwise. Retention only has to cover the window in which a retry can plausibly arrive, which is bounded by the client's own retry policy plus a margin — 24 hours is generous for a mobile client and 7 days is generous for a server-to-server integration with a job queue behind it. Keeping the table partitioned by day and dropping old partitions costs nothing and avoids the alternative failure, which is a 400-million-row table with a UUID primary key whose index no longer fits in cache and whose claim insert has quietly become the slowest thing on the request path.

What idempotency does not buy#

It is not exactly-once, the guarantee that an operation is performed one time and no more no matter how many attempts arrive. The second attempt in the fixed design still runs whenever it arrives before the first commits and after the first began — the 409 path costs a round trip and nothing else, but if you choose to let both generate and discard the loser, the money is still spent. Idempotency bounds the durable consequences, and dollars are not a durable consequence.

There are also callers you do not control, and for those the key has to be derived rather than supplied. A server-side fallback that hashes tenant, conversation, message content and a coarse time bucket will collapse a burst of identical requests within, say, a ten-second window, and it will also collapse a user who genuinely pressed send twice because nothing happened. That trade is acceptable for a burst window measured in seconds and unacceptable at minutes, and it should be recorded as a distinct key type on the record — key_source: 'derived' — so that the eventual investigation into "why did my second message vanish" has something to look at.

It also fails completely if the client mints a new key per attempt, which is the most common way this ships broken. The key is a contract between the client and the orchestrator, so the client SDK has to be the thing that generates it, and a server-side default that mints a key when the header is absent gives you a system that looks instrumented and dedupes nothing. Logging the proportion of requests arriving without the header is worth more than any server-side heuristic.

And it does nothing about the streaming interruption from the previous chapter. Those two failures are often described together because they showed up in the same incident, and they have opposite shapes: the interrupted stream is one operation whose effects were split across two calls, while the client retry is two operations that should have been one. Fixing either does not fix the other, and a system that has fixed both still cannot tell you whether any of the answers it committed exactly once were correct.