Retries duplicate side effects unless the key comes from the request

Finance found it during month-end: 340 customers had been refunded twice for the same order over a nine-week period, £15,300 in duplicate credits, none of it recoverable because clawing back a goodwill refund costs more in support contacts than the refund is worth. The trace on one of them ran like this.

14:02:11.402  agent run 7f3a  step 4  tool=issue_refund  args={"order":"A-88214","amount_p":4500}
14:02:11.418  POST https://payments.internal/v1/refunds  (attempt 1)
14:02:21.418  urllib3.exceptions.ReadTimeoutError: read timeout=10
14:02:21.930  POST https://payments.internal/v1/refunds  (attempt 2)
14:02:23.107  201 Created  refund_id=rf_9KQ2  amount_p=4500
14:02:23.109  agent run 7f3a  step 4  ok
--- payments service, same window ---
14:02:11.501  refund rf_8XM1  order A-88214  4500p  settled 14:02:22.884
14:02:23.088  refund rf_9KQ2  order A-88214  4500p  settled 14:02:34.010

The first attempt succeeded. It took 11.4 seconds to return, the client gave up at 10, and the retry that fired half a second later was correct behaviour by every rule the HTTP client had been given. Nothing about this is specific to language models — it is the oldest ambiguity in distributed systems, where a timeout tells you that no response arrived and says nothing about whether the effect happened. The property that resolves the ambiguity is idempotence: an operation is idempotent when performing it a second time has the same effect as performing it once, so a caller who cannot tell whether the first attempt landed is free to try again. Agent systems hit the ambiguity more often for two structural reasons: tool calls are slow enough that timeouts are common rather than exotic, and an agent framework will happily retry a failed step on its own, sometimes with the model regenerating the call, so the retry does not even go through the code path where you thought you put the guard.

Six places a duplicate comes from#

Before choosing where the key lives, it helps to know how many independent things are capable of replaying the same effect, because a guard placed at the wrong layer catches one of them and misses the others.

The HTTP client retries on timeout, which is the case in the trace. A proxy or load balancer in front of the service retries on a connection reset, invisibly to your code and without your headers changing, which is why some gateways must be configured not to retry non-idempotent methods. The queue that delivers work redelivers when a consumer fails to acknowledge in time, and every mainstream broker is at-least-once, so a worker that takes 35 seconds on a queue with a 30-second visibility timeout will process the same message twice as a matter of routine. The agent framework retries a failed step, sometimes by re-invoking the model, so a duplicate arrives with a fresh reasoning trace and, if the key is generated during planning, a fresh key. The user clicks the button twice, or refreshes a page that posts on load. And a scheduled job overlaps its previous run because the previous run has not finished.

Those six differ in whether they are visible to you and in where they can be intercepted, but they all present at the effect boundary as two requests that ought to be one, which is the argument for putting the guard at that boundary rather than trying to make each source well-behaved.

Retry policy interacts with this and should be written down explicitly rather than inherited from a library default. A connection error raised before any bytes were written is safe to retry with no guard at all, because nothing reached the server. A read timeout, a 502, a 503 and a dropped connection mid-response are ambiguous, and those are precisely the cases where the key does the work. A 400 or a 422 should never be retried, since the request will fail identically and the retries only consume rate limit. A 409 from an idempotency-aware service means a duplicate is in flight and the correct response is to back off and retry, which is not the same as failing the step. Encoding that classification in one place — a retryable(error) predicate — and using it everywhere prevents the common outcome where each tool wrapper makes a slightly different judgement and the aggressive one causes the incident.

The key has to come from upstream of the retry#

The fix has a shape that payment APIs settled on years ago. Stripe added the Idempotency-Key header in 2015 for exactly this case, and the pattern spread through PSPs, ledgers and messaging systems because the alternative — asking every caller to reconcile before retrying — does not survive contact with real clients. The server records the key, and a second request carrying the same key returns the stored result of the first instead of performing the effect again.

What makes it work or fail is where the key is generated. A key minted inside the retry loop is a new value on every attempt and deduplicates nothing, which is the single most common way this is implemented wrongly, and it is tempting because uuid4() next to the request builder looks obviously correct. The key has to be derived from the thing that stays constant across attempts, which is the originating request: the user message, the inbound webhook, the queue item that started this run.

The fourth row is the over-correction, and it is worth understanding before adopting it. Hashing only the arguments means the second legitimate refund of £45 on the same order — the customer had two problems, a month apart — is silently swallowed and returns the old refund ID, and the failure is invisible because the call succeeds. Including the run identifier scopes the deduplication to one agent run, which is the unit across which duplication is a bug and outside which repetition is a business decision. Where runs are long or the same step can legitimately recur inside one, adding a bucketed timestamp is the usual compromise.

Asking the model to produce the key belongs in the table because frameworks sometimes suggest it and it is worse than it looks. A model regenerating a tool call after an error will not reproduce a random identifier it emitted a minute ago, so the key changes exactly when you need it not to, and a model that has been prompt-injected can choose the key deliberately.

import hashlib, json

def idempotency_key(run_id: str, step: int, tool: str, args: dict) -> str:
    canonical = json.dumps(args, sort_keys=True, separators=(",", ":"))
    material = f"{run_id}|{step}|{tool}|{canonical}"
    return hashlib.sha256(material.encode()).hexdigest()[:32]

def call_tool(session, run_id, step, tool, args, attempts=3):
    key = idempotency_key(run_id, step, tool, args)   # computed once, outside the loop
    for i in range(attempts):
        try:
            r = session.post(f"https://payments.internal/v1/{tool}", json=args,
                             headers={"Idempotency-Key": key}, timeout=10)
            r.raise_for_status()
            return r.json()
        except (Timeout, HTTPError) as e:
            if i == attempts - 1 or not retryable(e):
                raise
            sleep(0.5 * 2 ** i)

The canonical serialisation matters more than it appears. Two dicts that differ only in key order, or in 4500 against 4500.0, hash differently and produce two refunds, so the serialisation has to be deterministic and the argument types have to be normalised before hashing. Amounts in integer minor units rather than floats removes an entire class of near-miss.

The server side is where the guarantee lives#

A header is a request for cooperation. The effect is only performed once if the service that performs it enforces that, and the enforcement is a uniqueness constraint plus a record of what was returned.

CREATE TABLE idempotency_records (
    key          text PRIMARY KEY,
    request_hash text        NOT NULL,
    status       text        NOT NULL DEFAULT 'in_progress',
    response     jsonb,
    created_at   timestamptz NOT NULL DEFAULT now()
);

The insert has to happen before the effect, not after, so that a concurrent duplicate loses the race rather than racing the ledger write. That leaves the in_progress case: a second request arrives while the first is still working, and the right response is a 409 telling the caller to retry later rather than a wait, because holding the connection open converts a duplicate into a thread leak under load. The request_hash column covers the other abuse, where a caller reuses a key with a different body — that is a client bug or an attack, and returning 422 rather than the stored response makes it visible instead of silently ignoring the new request.

Retention is a real operational decision. Twenty-four hours covers transport retries comfortably and costs almost nothing; long enough to cover a queue that can redeliver after a multi-day outage costs more, and a table that is never pruned becomes the largest thing in the database. Stripe keeps keys for 24 hours, which is a reasonable default to copy unless your redelivery window is longer than that.

Not every downstream will play along. Plenty of internal services and third-party APIs have no idempotency support at all, and for those the deduplication has to move into your side of the boundary: a local ledger row inserted under a unique constraint before the call, and a read-back after an ambiguous failure — list refunds for order A-88214 in the last five minutes and see whether one exists — before deciding to retry. The read-back is racy and imperfect, and it turns a certain duplicate into an unlikely one, which is the available improvement.

Watching the mechanism work#

A deduplication layer is a control that succeeds silently, which puts it in the category from earlier chapters where you cannot tell whether it is working or unnecessary. Two metrics make it visible. The replay rate — requests that matched an existing key and returned a stored response, as a fraction of all requests — should be small and non-zero; on the payments service it settled at about 0.4%, which is a direct measurement of how often the ambiguous-timeout case actually occurs and is far higher than most teams guess. The conflict rate, where a key arrives with a different request body, should be zero, and any non-zero value is a bug worth chasing immediately because it means two logically different actions are colliding on one key.

A spike in the replay rate is also a useful early alert on something else being wrong. When the payments service's p99 crossed the client's 10-second deadline during a database migration, the replay rate went from 0.4% to 22% within a minute, which surfaced the latency regression before the latency dashboard did, because every request was now completing after the client had already given up. That is the mechanism doing exactly its job — nobody was refunded twice — while signalling clearly that the system was in a degraded mode.

The other operational detail worth settling early is what a resumed agent run does. If a worker crashes at step 4 of 7 and the run is requeued, the replay of steps 1 through 4 should hit stored responses and cost nothing, which requires that the run identifier and the step index survive the crash — so they belong in the run's persisted state rather than in a Python object. A framework that regenerates the plan on resume, and therefore renumbers the steps, breaks this quietly: the keys differ, the effects repeat, and the trace looks like a perfectly reasonable agent doing its job twice. Checking that resumption reproduces identical keys is a five-minute test and it is the one that distinguishes a design that survives a deployment from one that only survives a happy path.

What idempotency does not cover#

It protects against duplicate delivery, not duplicate decisions. If the agent evaluates the customer's situation at 14:02 and decides on a refund, then evaluates again at 15:40 after the customer writes back and decides on a second refund, the keys differ because the run differs, both refunds are issued, and the system behaved exactly as designed. That is a planning defect, and the control for it is state the agent must check — a ledger of what has already been done for this order — rather than a key.

It is also not reversibility, and the distinction matters because the two get filed together as safety work. The hold-and-commit machinery from the previous chapter gives you a window in which nothing has happened; idempotency gives you the guarantee that when the commit finally runs, running it twice is the same as running it once. Neither one stops a correct-once refund that should never have been issued at all.

That case — the model was confident, the action executed cleanly and exactly once, and it was still the wrong action — is what the remaining controls exist for, and the first of them is putting a person in front of it.