Testing approval races and partial writes

An integration test here means one that runs the enforcement path against real infrastructure — a containerised database with real transactions, the real proposal store, the real policy engine — and fakes only what enforces nothing on your behalf. The reason to pay for that is specific to the failures this topic is about: an approval race needs two actors touching one record with elapsed time between them, a cancellation test needs a socket somebody is actually holding, and a partial write needs a failure part-way through a real sequence. The unit test that already exists will pass while every one of those is broken, at high line coverage, because coverage measures which lines ran rather than what they were run against.

Why the fakes agree with the bug#

A refund was approved for €82.40 and executed for €1,284.00. The tool layer had 94% line coverage, and a test named test_refund_requires_approval passed on every commit for eight months.

def test_refund_requires_approval(monkeypatch):
    store = FakeProposalStore()
    tool = RefundTool(store=store, payments=FakePayments())
    prop = tool.propose(order_id="ord_55192")
    with pytest.raises(NotApproved):
        tool.commit(prop.id)
    store.approve(prop.id, by="u_2290")
    result = tool.commit(prop.id)
    assert result.status == "refunded"

That test demonstrates that an unapproved commit raises and an approved one does not. It cannot demonstrate anything about the plan drifting, because FakeProposalStore returns whatever was stored and FakePayments refunds whatever it is asked to refund, so the re-plan and digest comparison from chapter 1, on pinned proposals, either exist or do not and the test is identical either way. The fakes were written by the same person who wrote the tool, at the same time, with the same mental model, which is why they agree with it. Every line of the digest check ran — against a fake whose state nobody changed.

Approval is the word in that test's name, and the test's verdict does not depend on whether the approval mechanism works. Delete the re-plan, delete the digest comparison, leave the raise-on-unapproved behind, and the test still goes green. That is the property to look for when judging any test that guards a control: run it in your head against a build with the control removed, and if the outcome is the same, the test is measuring something else. It is the same defect the earlier chapters kept finding in the controls themselves — a docstring that reads identically on the previewing call and the deleting call, a tenant comparison against a tenant the caller supplied — arriving one level up, in the thing that was supposed to catch them.

The coverage figure is worth taking seriously rather than dismissing, because it is not lying about anything. Ninety-four per cent of lines did execute, and the digest lines were among them. What line coverage records is which statements the interpreter reached; what decides whether a digest comparison means anything is the state it was compared against, and a comparison of a stored value with itself executes exactly the same statements as a comparison that would have caught the drift. So the metric is accurate and the inference drawn from it is not, and the gap between the two is precisely the class of bug this chapter is about. There is no coverage threshold that closes it, because the missing ingredient is a second actor and elapsed time rather than a line.

The interleaving is the test#

What makes the refund bug reachable is that two actors touched the same record with real elapsed time between the check and the use, and that is not a property any single-process test with mocks can exhibit. The integration test that finds it needs a real database with real transactions, the real proposal store, the real policy engine from chapter 2, on scoped allowlists, and a controllable stand-in only at the outermost boundary — the payment provider — where a programmable HTTP fake lets you dictate latency and failure.

Written out, the test is short, and almost all of its value is in the assertions after the exception:

@pytest.mark.integration
def test_approval_does_not_survive_a_change_to_the_record(db, agent, payments_fake):
    order = db.insert_order(id="ord_55192", total="82.40", card_last4="4412")
    prop = agent.call("propose_refund", {"order_id": order.id})
    agent.approve(prop["proposal_id"], by="u_2290")

    # A second actor, on its own connection, in its own transaction.
    with db.other_connection() as conn:
        conn.execute("UPDATE orders SET total = '1284.00' WHERE id = %s", (order.id,))

    with pytest.raises(PlanDrifted) as err:
        agent.call("commit_proposal", {"proposal_id": prop["proposal_id"]})

    assert err.value.diff == {"total": ["82.40", "1284.00"]}
    assert payments_fake.calls == []                       # nothing was charged
    assert db.refunds_for(order.id) == []                  # and nothing was written
    events = audit.for_run(agent.run_id)
    assert events[-1]["outcome"] == "refused"
    assert events[-1]["approval"]["plan_digest"] == prop["plan_digest"]

The payments_fake.calls == [] assertion is the one that would have caught the production bug, because a version of this system that refunds first and compares digests afterwards passes every other line. The audit assertion matters for a different reason: it checks that the refusal was recorded with enough context to explain itself, which is the property chapter 4, on the tool execution audit log, said gets discovered missing during an incident.

The diff in that assertion is also the whole explanation of the production incident, which is a property worth noticing because it is rare. The order's total went from eighty-two forty to one thousand two hundred and eighty-four, and the difference between those two figures is the combined total of the three orders that were merged into it — a bit over twelve hundred euros arriving as one consolidation. Nothing else has to be invoked. The refund tool was not wrong about the order, the payment provider was not wrong about the amount, and no argument was mis-derived by a model; the record simply meant something different at the second read than at the first, and the two totals reconcile with the merge and leave no remainder. That is why the digest is the right instrument here rather than, say, a bound on the refund amount. A cap would have caught this particular case and would say nothing about a merge that happened to leave the total smaller.

Backend engineers have a name for the underlying defect from a different context. A digest over the plan is optimistic concurrency control — read a version, do some work, compare the version at write time, abort if it moved — and the failure the naive design has is the same lost-update anomaly, in which one writer's change is quietly overwritten by a second writer that read the same now-stale value, you get from SELECT followed by UPDATE without a version predicate or a FOR UPDATE lock. The wrinkle agents add is the duration. Ordinary optimistic concurrency spans milliseconds inside one request, whereas an approval window spans however long a human takes to read a Slack message, and a conflict window measured in minutes converts an anomaly that occurs rarely into one that occurs weekly.

The step from "rarely" to "weekly" is a multiplication, and doing it is what makes the argument transferable to your own system. For a fixed rate of independent writes against a record, the chance that one lands inside a given window grows in proportion to the length of the window. An ordinary read-modify-write inside a single request leaves a window of milliseconds. The refund left a window of four minutes, which is a few hundred thousand times longer, so a collision that would have been a once-a-decade curiosity at request scale arrives at a rate several hundred thousand times higher. Nothing about the mechanism changed and nothing about the code was novel; only the exponent on the clock moved. Two things follow. Anything that lengthens the window — a human in a meeting, an agent that does four minutes of other work between the approval and the commit, a queue backing up — is a security change and should be treated as one. And a system that has run a consolidation workflow for two years without ever hitting this has not demonstrated that it is safe; it has demonstrated that nothing had previously held a read open for minutes.

There is a class of drift the digest does not catch, and it is worth knowing where the mechanism stops. If the plan is expressed over identifiers and the meaning of an identifier changes while the values stay identical — the order total is still €82.40 but the order has been reassigned to a different customer, or the Slack channel with the same id has been converted to a shared external channel — the digest matches and the commit proceeds. Covering that means including the fields the decision actually depended on in the plan, not only the ones being written, which is a judgement call the test suite can then pin down: add customer_id to the digest and add a test that mutates it.

Where to draw the boundary between real and fake is the decision that determines whether the suite is worth its runtime. The rule that holds up is that everything enforcing a safety property must be real, and everything beyond the last enforcement point may be faked. The database is real, because row locks and transaction isolation are enforcement; a container started per test session with the production schema applied through the same migrations costs about four seconds of startup and removes an entire category of false confidence. The policy engine is real, because it is the control under test. The payment provider is faked, because it enforces nothing on your behalf, but the fake needs its own discipline: a stub whose response shape drifts from the provider's is a test that passes against a system that would fail in production, so the fake should be generated from the provider's OpenAPI description, the machine-readable specification of its endpoints and payloads, or pinned by a recorded contract that is refreshed on a schedule.

Runtime is the usual objection, and the numbers are manageable if the suite is kept to the scenarios that only integration can reach. Eleven such tests against a containerised Postgres and two HTTP fakes run in about 40 seconds on a developer machine, which is per-pull-request territory. What pushes a suite past that is using it for coverage of ordinary tool logic that unit tests handle in milliseconds, so the discipline is to keep the integration suite small, name each test after the failure it prevents, and require that a new test in it comes with the incident or the near-miss that motivated it.

The two numbers in that paragraph pull in opposite directions and it is worth seeing how. The container start is about four seconds and it is paid once for the whole session, not once per test, because the schema is applied and the container is reused. Take it off the forty and what is left is a little over three seconds for each of the eleven tests, which is the cost of real transactions, real network round trips to two fakes, and real elapsed time between two actors. That per-test figure is the one that scales, and it is roughly a thousand times what a unit test costs. So the shape of the trade is fixed by arithmetic rather than by taste: a fixed cost you pay once and would happily pay, and a marginal cost that makes a hundred integration tests a coffee break. Keeping the suite to the interleavings only integration can reach is not asceticism, it is the only configuration in which the marginal cost stays invisible.

Making the interleaving deterministic#

The temptation in a test like the one above is to reach for time.sleep to get the ordering you want, and a suite built that way is slow and flaky in roughly equal measure. Two techniques keep these tests deterministic.

For orderings inside one process, use an explicit barrier the production code exposes only in tests — typically a no-op hook that the test replaces. The commit path calls hooks.before_replan(), which is empty in production and, in the test, releases the second actor and waits for it to finish.

def test_race_between_replan_and_write(db, agent, hooks):
    prop = agent.call("propose_refund", {"order_id": "ord_55192"})
    agent.approve(prop["proposal_id"], by="u_2290")

    def mutate_now():
        with db.other_connection() as conn:
            conn.execute("UPDATE orders SET total='1284.00' WHERE id='ord_55192'")
    hooks.before_write = mutate_now      # fires after the re-plan, before the charge

    with pytest.raises((PlanDrifted, SerializationFailure)):
        agent.call("commit_proposal", {"proposal_id": prop["proposal_id"]})

That second test is the sharper one, because it exercises the window between the re-plan and the write, which a digest comparison alone does not close. Closing it needs either a row lock held across the check and the write, or a version predicate in the update statement itself, and the test tells you which of the two you have.

For timing, inject the clock rather than sleeping. The deadline object from chapter 3, on task budgets, should take a time source, so a test can advance it by 9,001 milliseconds instantly and assert that a call is refused for budget rather than waiting nine seconds on every run.

The three other scenarios that belong in the same suite#

A forced timeout test exists to prove that cancellation reaches the work, which is the claim chapter 3 said most implementations cannot make. The payments fake is configured to accept the connection, hold it for 30 seconds and then respond, and the assertions are about what happened on both sides.

def test_timeout_cancels_the_downstream_request(agent, payments_fake, clock):
    payments_fake.delay_seconds = 30
    prop = agent.call("propose_refund", {"order_id": "ord_88104"})
    agent.approve(prop["proposal_id"], by="u_2290")

    with pytest.raises(ToolTimeout) as err:
        agent.call("commit_proposal", {"proposal_id": prop["proposal_id"]})

    assert err.value.state == "unknown"                    # not "failed"
    assert payments_fake.observed_client_disconnect is True
    assert payments_fake.request_count == 1                # no silent retry
    assert audit.for_run(agent.run_id)[-1]["duration_ms"] < 9500

observed_client_disconnect is the assertion that distinguishes a real cancellation from a caller that merely stopped waiting, and it is why the fake has to be a process holding a socket rather than a mock returning a future. For subprocess-backed tools the equivalent assertion is that the process group is gone: spawn a command that writes its own children's pids to a file, time it out, then check that none of them are alive.

A denial test runs the whole path with a session belonging to one tenant and a resource belonging to another, exactly as chapter 2 described, except that here it goes through the real HTTP tool endpoint rather than calling authorize directly. The additional thing it checks is what the agent does with the denial: that the error surfaced to the model is a structured refusal it can report rather than a stack trace it will paraphrase into "I encountered a technical issue", and that the run does not retry the denied call in a loop, which is a failure mode that shows up as a thousand identical policy denials in ninety seconds.

A bypass test is the least common and the most valuable. It asks whether there is any path from model output to a write that does not pass through the proposal. Three concrete cases cover most of it: committing a proposal that was never approved, committing one that the agent's own principal approved, and a tool result containing injected text instructing the model to call the write tool directly with dry_run semantics disabled. That last one is a real test with a real assertion — the model may well try, and what you are asserting is that the attempt is refused by the server and recorded, not that the model declines. Alongside those, a structural test over the tool registry is cheap and catches the regression that matters most:

def test_every_mutating_tool_has_a_proposal_path():
    for name, spec in TOOL_REGISTRY.items():
        if spec.mutates:
            assert spec.proposal_of is not None, f"{name} writes without a proposal"
            assert "dry_run" not in spec.parameters, f"{name} exposes a model-set dry run"

Partial writes, when there is nothing to roll back#

The bulk price update from chapter 1 fails at item 700 of 1,240 with a connection reset. Inside one database that is a transaction and the answer is a rollback, but the plan also fires 1,240 search-index webhooks, pushes a partner feed and sends 8,934 emails, and none of those are transactional. Rolling back the database leaves the emails sent.

The test that finds this configures the fake to fail at a specific index and asserts a property rather than an outcome, because either answer can be correct depending on the design. If the tool is transactional over the database and defers side effects through an outbox, a table of pending effects written inside the same transaction and drained afterwards by a separate worker, the assertion is that the price table is unchanged and zero webhooks were delivered. If the tool applies item by item with a per-item idempotency key, the assertion is that exactly 699 items are updated, that the progress is recorded against the proposal, and that re-committing the same proposal resumes at 700 and does not touch the first 699 again. What must never pass is the third possibility, in which 699 items are written, the proposal is marked failed, and a retry re-applies the 15% reduction to items that already have it, which is how a €89 jacket ends up at €64.

Both numbers in that paragraph are checkable against the failure, and getting either wrong in the test is how a suite passes against a broken design. The count is one less than the failing index: item seven hundred is the one that failed, so six hundred and ninety-nine completed, and a test asserting seven hundred is asserting that the item which raised the connection reset was nevertheless applied. The price is the multiplication done twice. Take fifteen per cent off eighty-nine euros and the jacket is at seventy-five sixty-five, which is the intended outcome and the figure the preview in chapter 1, on dry runs and pinned proposals, showed a reviewer. Take fifteen per cent off that and it lands a little over sixty-four, which is the number in the sentence above. The compounding the preview was built to expose has arrived by a different route entirely — not from a rule applied to the wrong base, but from a correct rule applied twice to the same row — and no preview can catch it, because the second application happens after the human has gone. That is the argument for a per-item idempotency key rather than for a better preview: the reviewer's instrument works on the plan, and this defect is created during the execution.

Two limits are worth stating. These tests demonstrate that a specific interleaving is handled, and they say nothing about interleavings you did not think of; property-based testing over operation orderings, which generates orderings mechanically and asserts that an invariant survives all of them, and fault-injection runs, which fail dependencies on purpose against a staging deployment, cover more ground and are a larger investment. And a suite that fakes the human approver has to be watched, because the fake approver is the easiest place for the self-approval check to be quietly disabled — the test helper calls store.approve(id, by="u_2290") directly, and one refactor later u_2290 is the agent's own service principal and nothing fails.

The suite that earns its runtime is therefore small and named after failures rather than after functions: one test per interleaving a control exists to survive, each asserting on what the dependency saw as well as on what the caller got back. It is not a proof that the controls hold. It is the mechanism that notices when a refactor removes one, which is how most of these controls actually stop working.

All of this establishes that the controls work. It does not establish that they will still be there in three months, and the pressure that removes them does not come from an attacker.