One logical call becomes two at five different layers

"Something retried below me" is a diagnosis without an address. It is true of every duplicate, it suggests no action, and it is compatible with five different mechanisms whose fixes have nothing to do with each other. Turning it into an address is the work of this chapter, and the work is finite because the list of things that can execute one logical call twice is short enough to memorise.

The reason the address matters is narrower than "you should know how your system works". An idempotency key — the caller-supplied string that makes a provider return the first request's result instead of creating a second thing — is only worth anything if the two attempts carry the same one. Whether they do depends entirely on where the second attempt was born, because each layer re-executes from a different starting point and therefore preserves a different set of identifiers. Mint the key from something the responsible layer does not preserve and you have written the failure again, this time with a header on it.

The five places a duplicate can be born#

The candidate set is short enough to enumerate from memory, which is the point of enumerating it. Each entry fires under different conditions and leaves a different mark.

Read the third column by its gaps rather than by its entries, because the gaps are what make the table usable at three in the morning. The layers do not merely differ, they differ by roughly an order of magnitude in the time between attempts each time you move down a row. An SDK retrying a connection reset fires within its own backoff, which is seconds. A step retry usually follows a client-side timeout, so the gap is the timeout — tens of seconds. A redelivery waits for a visibility timeout or an eviction to be noticed, which is minutes. Those separations exist because each layer's constant was chosen for a different purpose by a different set of authors, and the accident is a useful one: a measured gap lands in exactly one band. If two of these layers had been configured with similar constants they would be indistinguishable from timing alone, and the diagnosis would need the fingerprints in the third column instead.

The final column is the one that decides the fix, and reading down it shows why a single mitigation can cover the first four and cannot cover the fifth. The first four are all re-executions of a call the model asked for once, so there exists an identifier that both attempts inherit. The fifth is the model asking a second time, and there is no sense in which the two requests are the same call.

That column is a closing identity worth stating as one, because it licenses a specific and limited claim. What it licenses: if a value can be found that is carried by all four of the top rows, then one derivation defends against four independent mechanisms without any of them being configured, detected or even identified. What it leaves over is not small. It says nothing about how often the fifth row fires relative to the other four, so "covers four of five mechanisms" is a floor on what the derivation is worth and not an estimate of it. A system whose duplicates are mostly model re-emissions would gain almost nothing from the work in this chapter, and the only way to know which system you have is to look.

Why the failing attempt left no span#

The 30-second-hole incident from the previous chapter was the third row, and the reason the tool span for the first attempt was missing is worth spelling out because it is a defect in the instrumentation rather than in the runtime. The span callback opened a span in on_tool_start and closed it in on_tool_end, and the first attempt did not end — it raised a timeout, which fires on_tool_error.

Follow that through the exporter's rules and the hole is not a mystery at all. A span becomes a record when it ends; ending is what hands it to the batch processor, and the processor is what sends it. A span that is started and never ended is a live object in a dictionary, so it is never handed over, never exported, and never seen. The first attempt therefore produced a span that existed in memory for the lifetime of the process and in the backend not at all. Instrumentation that only handles the success path goes blind at precisely the moments you need it, and the correction is four lines.

from opentelemetry.trace import Status, StatusCode

    def on_tool_error(self, error, *, run_id, **kw):
        span = self._open.pop(run_id, None)
        if span is None:
            return
        span.record_exception(error)
        span.set_status(Status(StatusCode.ERROR))
        span.set_attribute("tool.attempt_failed", True)
        span.end()

The pop with a default and the early return are not defensive padding. A callback can fire for a run the handler never opened a span for — a tool that failed before dispatch, or an error raised during a shutdown — and popping without a default would raise inside the error handler, replacing an observable failure with an unobservable one. The general shape is that error callbacks run under conditions the success path never sees, so they have to tolerate a state the success path guarantees.

Evidence you did not write yourself#

The instrumentation lesson generalises: your own telemetry is written by people who were imagining the working case, so the decisive evidence in a duplication incident usually comes from a system you do not control. Three sources settle most of these questions in under ten minutes, and each of them is authoritative about something your own records cannot be authoritative about.

The payment provider's request log is the first, because it is the only ledger that counts executions rather than intentions. That distinction is the whole value of it. Your log counts the times your function decided to record something; the provider's counts the times a request arrived. Those are different quantities over different populations, and every duplicate incident is a disagreement between them, so a source that measures the second one is not merely more evidence — it is the only side of the comparison you did not write. Every serious API keeps request ids, timestamps and the idempotency key each request carried, and the shape of what it returns tells you which layer you are in: two requests two seconds apart with identical bodies and no key is an SDK or gateway retry; two requests thirty seconds apart is a client timeout followed by a step retry; two requests six minutes apart is redelivery.

stripe charges list --customer cus_8123 --created.gte 1717422000 \
  --expand data.request \
  | jq '.data[] | {id, created, request: .request.id, idem: .request.idempotency_key}'
# {"id":"ch_3Qa1RtK","created":1717424527,"request":"req_8Kx2","idem":null}
# {"id":"ch_3Qa1RwB","created":1717424568,"request":"req_8Kx9","idem":null}

Two distinct request ids in that output do more work than the two charge ids do. Two charges could in principle come from one request that the provider processed twice; two request ids say the provider was asked twice, which places the duplication on your side of the network and rules out an entire class of explanation before any of your own logs are opened.

The checkpoint table is the second, and it answers the redelivery question directly, because a resumed run reads a checkpoint it did not write in that invocation. Querying it by thread id gives you the runtime's own account of which steps it believed had completed and when.

select checkpoint_id, metadata->>'step' as step, metadata->>'source' as source, created_at
from checkpoints
where thread_id = 'run_9f21c0'
order by created_at;
--  step | source | created_at
--     4 | loop   | 14:21:59.204
--     5 | loop   | 14:28:11.887   <- written six minutes later, by a second worker

What makes that decisive is the gap between two consecutive rows rather than either timestamp. Steps of one invocation are written seconds apart, because they are separated by one model call and one tool call. A gap of minutes between consecutive steps means nothing was running in between, which means the work stopped and started again, which is redelivery and not a retry inside a live invocation. The runtime keeps this record for its own purposes, so it exists whether or not anybody instrumented anything.

The queue's own metadata is the third. SQS exposes ApproximateReceiveCount, most brokers expose a delivery count, and a value above one on the message that carried the run is conclusive. It costs nothing to copy that value into a span attribute at the top of the worker, and doing so turns a ten-minute investigation into a filter on a trace search.

Derive the key, do not generate it#

Once the layer is known, the placement of the idempotency key follows from the last column of the table, and the general rule that covers four of the five rows is to derive the key from the model's tool call rather than generate it at the moment of the HTTP request. Every tool call the model emits carries an identifier — toolu_01WkP9Qm in the Anthropic message format, call_8Kx2 in the OpenAI one — that is assigned once, stored in the assistant message, meaning the model's own turn in the conversation, and therefore present in the checkpointed state. Anything downstream that re-executes the call re-reads that same id, because they all start from the same message list.

That last sentence is the whole mechanism and it is worth taking slowly. A step retry re-enters the node; the node reads the message list; the message list is the checkpointed state. A redelivery starts a new process; the process loads the checkpoint; the checkpoint contains the message list. In both cases the path back to the tool-call id is the same path, and the id was written into it once, by the model's turn, before any of the duplication happened. A uuid4() generated inside the tool function has none of that property; it is regenerated on every attempt, which is the failure mode dressed up as the fix.

import hashlib

def idem_key(thread_id: str, tool_call_id: str) -> str:
    """Stable for one logical tool call. Survives node re-entry, redelivery and resume."""
    return hashlib.sha256(f"{thread_id}:{tool_call_id}".encode()).hexdigest()[:32]


def dispatch(state, tool_call) -> dict:
    key = idem_key(state["thread_id"], tool_call["id"])
    result = TOOLS[tool_call["name"]](**tool_call["args"], idem_key=key)
    return {"tool_call_id": tool_call["id"], "content": result}

Check that function against the two conditions a key has to meet. It must be identical across a duplicate: both inputs come from state that a duplicate re-reads rather than regenerates, so it is. It must differ across genuinely distinct calls: the tool-call id is assigned per call by the model, so two real charges get two ids and two keys. The thread id is in there to keep keys from colliding across runs that happen to reuse a tool-call id, which costs nothing and removes a whole category of question. Hashing at the end is cosmetic — it produces a fixed-width string a provider will accept — and carries no security property, since nothing here is secret.

Deriving rather than generating is what makes the key survive each layer without any layer knowing about it. A step retry re-enters the node with the state it had on entry, so tool_call["id"] is unchanged and the key comes out the same. A redelivery resumes from the checkpoint that contains the same assistant message, so the key comes out the same in a different process on a different host. An SDK or gateway retry never reaches this code at all, since it replays the request the tool already built, headers included. One line of derivation covers four independent mechanisms, and it does so without a database, a lock, or a set in module scope.

The key also has to reach the wire, which is a separate check worth doing explicitly rather than assuming. Passing idempotency_key= to an SDK method usually sets an Idempotency-Key header, but some clients only apply it to a subset of endpoints, and some proxies strip unknown headers. Confirming it takes one look at the provider's request log after a deliberate retry: the key column should be populated and identical on both rows. In the output above it was null on both, which is what a missing key looks like from the only vantage point that matters.

The duplicate an idempotency key cannot see#

The fifth row stays broken under all of this, and it is worth being precise about why rather than treating it as an edge case. When the first charge times out and the tool returns an error to the model, the model reads that error, concludes the charge did not happen, and emits a second charge_subscription call with a fresh id. The derived key differs because the tool call genuinely differs, the provider sees a new request, and the customer is charged twice by a system in which every layer behaved exactly as designed. No retry fired anywhere.

Notice that the key is working correctly there. It is required to differ across genuinely distinct calls, and by the only definition available to it these are two distinct calls: two ids, two decisions, two turns. The information that they refer to one intended charge exists nowhere in the message list. Asking the key to catch this would be asking it to violate the second of the two conditions that make it safe, which would trade a duplicate for a dropped payment.

The mitigation is therefore not an idempotency key but the content of the error the tool returns. A timeout on a request that creates something is not a failure; it is an unknown outcome, and telling the model that a charge failed is a false statement about the world. A tool that returns {"status": "unknown", "detail": "request timed out after 30s; the charge may have been created", "next_step": "call get_charge_status before retrying"} gives the model something it can act on correctly, and pairing that with a get_charge_status tool that looks up by the derived key closes the loop. This is prompt and tool-surface work, not runtime work, which is the recurring shape of the boundary: the framework can make a call happen twice, and only your tool contract can make the second one harmless.

Retention is the other limitation to know about, and it bounds every fix in this chapter rather than only the fifth row. Providers store idempotency keys for a bounded window — Stripe's is 24 hours, others range from one hour to seven days — so a key protects against a retry inside that window and against nothing outside it. A run that a human pauses for two days and resumes will re-execute cleanly as far as the provider is concerned, because the key has expired and the request looks new. The protection is a property of a time-limited record on somebody else's side, and an agent whose runs can outlive that record has a gap that no amount of care about key derivation closes. Long-running agents with human pauses need a durable record of their own effects, which is a different mechanism from an idempotency key and a considerably more expensive one to build.

Both of the cheap fixes here are cheap only once the layer is identified, and neither of them was available from the framework's documentation, because both depend on internal behaviour — what its retry policy considers retryable, when its callbacks fire, and what its checkpointer writes and when. Working that out from the outside, by running the same agent against a tool that fails on purpose, is one way to find out.

Building the piece yourself is the other, and it prices the framework more honestly than any evaluation matrix. The durable-record mechanism the last paragraph named is the obvious candidate, since it is the one the derived key cannot substitute for, and writing it turns out to price several other things at the same time.