Retries turn one attempt into several hook firings

The quarterly reconciliation is a two-column report. On the left, the payments ledger: 8,109 completed transfers between 1 January and 31 March. On the right, the agent platform's audit table, which is supposed to carry one row per transfer_funds the agent ever attempted: 8,412 rows. The auditor's email asked two questions, and only the first was the obvious one.

Three hundred and three of the audit rows were duplicates — same amount, same source and destination accounts, same recipient reference, timestamps between 400 milliseconds and 31 seconds apart, one transfer in the ledger. That accounted for 8,412 minus 303 equals 8,109, which is where an engineer stops reading and closes the ticket. The auditor had not stopped there. One row in the audit table, 2025-02-19T11:04:07Z, €7,812.00 to a supplier account, had no counterpart in the ledger at all, and the arithmetic only balanced because a second row elsewhere was a duplicate that happened to cancel it out.

The audit hook had been unchanged for a year. What had changed was release 4.19 of the internal tool client, whose changelog entry read add retry with jitter on 5xx and timeouts (max 3 attempts).

That release was not a mistake. The payments gateway returned a 5xx or timed out on roughly 0.4% of requests, and before retries were added, one failed request usually ended the agent's run, because the model would report the transfer as failed and stop. Retries took end-to-end task completion from 91.3% to 99.2% over the following month. The team that shipped it had no reason to think about a hook they did not own, and the team that owned the hook had no reason to read a client changelog. Nothing in either codebase records that the audit table's correctness depended on each call being delivered exactly once.

One logical call, several deliveries#

The hook was registered before the tool call, and the harness fires it before each delivery, so a retried call fires it again. Nothing in the hook's input distinguishes the second delivery from the first.

That trace produces two audit rows and one transfer, which is the 303. The orphan row comes from the mirror image of it: the hook wrote its row, the client sent the request, the payments API rejected it on a validation error after the agent's session had already been interrupted, and no retry followed. The audit table recorded an attempt that never became a transfer, which is precisely what it was designed to record and precisely not what everyone reading it believed it meant.

The two symptoms have one cause, which is that the table conflated two different facts under one row. A pre-call hook is in a position to assert exactly one thing: at this moment, this call was proposed with these arguments. It cannot assert that the call executed, that it succeeded, or that it executed once, because all three of those are facts about a future it does not observe. Every schema that stores an attempt and is read as an outcome carries this bug latently, and a retry policy is only the most common thing that wakes it up. Network partitions, harness restarts mid-turn, and a user pressing escape and re-asking do the same.

A stable identifier for the attempt, not for the delivery#

The fix has two halves, and the first is finding something that is the same across deliveries of one logical call. A UUID generated inside the hook is exactly wrong, since the hook runs again and generates a new one. The harness supplies a candidate: the event payload carries a tool-use identifier that is assigned when the model produces the call and is stable across the client's retries of it.

#!/usr/bin/env python3
import hashlib, json, sys, psycopg

event = json.load(sys.stdin)
if event["tool_name"] != "transfer_funds":
    sys.exit(0)

args = event["tool_input"]
attempt_id = event.get("tool_use_id") or hashlib.sha256(
    json.dumps(
        [event["session_id"], event["tool_name"], args],
        sort_keys=True, separators=(",", ":"),
    ).encode()
).hexdigest()

with psycopg.connect(DSN, connect_timeout=2) as conn:
    conn.execute(
        """
        INSERT INTO transfer_attempts
          (attempt_id, session_id, amount_cents, dest_account, ref, proposed_at)
        VALUES (%s, %s, %s, %s, %s, now())
        ON CONFLICT (attempt_id) DO NOTHING
        """,
        (attempt_id, event["session_id"], args["amount_cents"],
         args["dest_account"], args.get("reference")),
    )
sys.exit(0)

The ON CONFLICT DO NOTHING is what makes repeated delivery collapse, and it is worth being clear that the identifier is doing the work rather than the clause. The fallback hash exists because not every harness or wrapped client supplies a tool-use id, and it has a real weakness: two genuinely distinct transfers of the same amount to the same account with the same reference inside one session hash to the same value and collapse into one row. Adding a monotonic counter to the hash input would break the collapse across retries, which is the property you were buying, so the honest resolution is to make the caller's payload carry something that distinguishes the two — which is what an idempotency reference on a payments API is for, an identifier the server remembers so that a request it has already carried out returns the original result instead of moving the money a second time — and to treat the hash as a degraded mode you monitor rather than a design.

The second half is recording the outcome as its own fact, at the boundary that can see one. Chapter 2's table says which boundary that is, and the row it writes is keyed by the same identifier.

# PostToolUse: outcomes, keyed by the same attempt_id
result = event.get("tool_response", {})
conn.execute(
    """
    INSERT INTO transfer_outcomes (attempt_id, status, provider_ref, observed_at)
    VALUES (%s, %s, %s, now())
    ON CONFLICT (attempt_id) DO UPDATE
      SET status = EXCLUDED.status, provider_ref = EXCLUDED.provider_ref
    """,
    (attempt_id, "succeeded" if result.get("ok") else "failed",
     result.get("provider_ref")),
)

After the split, the February reconciliation runs as a join, and the rows that used to be invisible become the report's most interesting output: attempts with no outcome. There were 13 of them across the quarter, of which 11 were sessions a user had interrupted, one was a harness crash, and one was the €7,812 validation failure that had been sitting in the audit table dressed as a completed transfer. An auditor can act on a list of thirteen unresolved attempts. Nobody can act on a count that happens to balance.

The same arithmetic breaks anything a hook accumulates#

Audit rows are the easiest case to reason about because a duplicate is visible later. Hooks that accumulate state to make a decision fail harder, because the duplicate changes the decision itself.

Every row is the same mistake in a different costume: the hook treats a delivery as an event in the world when it is an event in a transport. The lock-file case is worth dwelling on because its failure looks like the hook working. The agent proposes a call, the hook takes the lock, the request times out, the client retries, the hook sees a lock held by "someone", and denies. The transcript shows a hook correctly preventing concurrent writes, and the only clue is that the concurrent writer was the same call.

Nested execution produces a related but distinct duplicate. A parent agent delegating to a subagent generates a hook firing at the parent's boundary when it launches the subagent and further firings inside the subagent for each tool call it makes, and if your counter increments on both, one unit of work is counted twice. That is not repeated delivery, it is two genuine events at two levels, and the fix is different: decide which level owns the fact and make the other level not write it. Attempting to deduplicate them by attempt id will not work, because they legitimately have different ids.

What being idempotent in the hook does not buy you#

The hook is now idempotent, meaning a second delivery of the same call leaves the world exactly where the first delivery left it. The transfer is not. Nothing above prevents the payments API from committing €7,812 twice when the client retries a request that had already succeeded, and the trace at the top of this chapter is a diagram of exactly that risk. End-to-end idempotency is a property of the tool and its backend, established by the caller supplying a key the server deduplicates on, and it lives outside the hook's reach.

What the hook can do is participate. If the payments tool accepts an idempotency_key and the model does not supply one, a hook that fills it from the attempt id is a legitimate normalisation by the test in chapter 3: the field is one the caller neither set nor reads, and no correct continuation by the agent differs depending on whether it was added. That is a narrow and useful role — the hook carries a stable identity into a system that needs one — and it is not the same as the hook making the operation safe.

There is also a class of interruption no identifier survives. If the hook process is killed by its own timeout after the INSERT has committed but before it exits, the harness sees a failed hook and may block the call, leaving an attempt row for a call that was never sent. Ordering the work so the side effect happens last narrows the window without closing it, and the durable answer is the reconciliation job that looks for attempts with no outcome older than some threshold and marks them unknown rather than deleting them. An audit trail that admits uncertainty is worth more than one that has been tidied into consistency.

Everything so far has assumed the hook's predicate has an answer: is a plan attached, is this path under infra/prod/, is this attempt already recorded. Rules arrive that look identical in shape and are not.