Testing unattended recovery

Every claim in this topic is a claim about what an agent does when it receives a particular result — that it corrects the named field rather than resampling all eight, that it stops when told a failure is terminal, that a replayed success is not committed twice. Claims of that shape cannot be settled by reading the code. They are settled by running the agent against the tool and looking at what it emits.

The test that does this is not the one most teams already have. Checking the final state answers "did the workflow complete", and completion is exactly the metric that a bare error string leaves untouched: the model guesses, reads, guesses again, and gets there. A test of recovery has to make claims about the path, which means it needs the trace, a bad call that a model would really emit, and a bound on attempts that came from somewhere other than the current behaviour.

Why completion is insensitive to the error body is worth having as a property rather than as an observation about one team's test. A completion assertion reads the store at the end and compares it against an expected value. Nothing in that comparison is a function of how many attempts were made, what each attempt cost, or what any of the failing responses said. So two implementations that differ only in their error path — one returning frames, one returning a mapped object — produce identical values for that assertion, and a test built on it cannot tell them apart by construction. The test is not weak. It is measuring a quantity the change does not move.

What the test has to assert, and what it has to forbid#

Four things go in it. The trace must contain at least one tool result that was an error, or nothing was recovered from. The terminal result must be correct against the underlying store rather than against the agent's own summary, because an agent that reports success it did not achieve is a distinct and common failure. The number of attempts must be bounded by a number chosen in advance. And the tool under test must be the production dispatch path, with no test-only normalisation, coercion, retry wrapper or repair layer between the model's arguments and the validator — which is a property you assert on rather than assume, because a repair layer is invisible from the test file.

Unattended is the load-bearing word in all of it, and the assertion that enforces it sits at the bottom of the test rather than the top. A recovery a human nudged is not the property being claimed, and the nudge can be very small — a clarifying message, a re-prompt, a retry someone triggered by hand. Checking that no human message entered the trace after the run began is what rules those out. Without it a suite can stay green while measuring a workflow that only ever completes because somebody was watching it, which is the exact failure the test was written to exclude.

import pytest
from harness import run_agent
from tickets import repo, dispatch

MAX_ATTEMPTS = 4

@pytest.mark.eval
@pytest.mark.parametrize("run", range(200))
def test_recovers_from_status_casing(run: int, seeded_ticket: str) -> None:
    assert dispatch.normalisers == [], "no repair layer may sit in front of validation"

    trace = run_agent(
        instruction="The customer says this is resolved. Close ticket T-8841.",
        tools=dispatch.production_tools(),
        max_attempts=MAX_ATTEMPTS,
        temperature=1.0,
    )

    calls = trace.calls_to("update_ticket")
    assert any(c.result.get("error") for c in calls), "the error path never ran"
    assert calls[-1].result.get("status") == "closed"
    assert repo.get("T-8841").status == "closed"          # the store, not the summary
    assert len(calls) <= MAX_ATTEMPTS
    assert trace.human_messages_after_start == 0          # unattended

The parametrised repetition is there because the thing being measured is a random variable. A single run of an agent at temperature 1.0 — the sampling setting the product itself runs at, so no two runs emit quite the same arguments — tells you about one sample from a distribution whose tail is where the incidents live, and a workflow whose tail is 8% shows it one run in twelve. Two hundred runs is enough to see a 5% tail clearly and enough to put a defensible number on the 95th percentile; the cost, at roughly 3,000 tokens per failing run, is about 0.6 million tokens and four minutes wall clock with a concurrency of twenty.

Both of those cost figures are one multiplication and one division, and having them settles the usual objection, which is that repeating a case two hundred times is expensive. Two hundred runs at roughly 3,000 tokens each is 0.6 million tokens, a quantity that at input prices never enters the argument. The wall clock is the run count over the concurrency: 200 runs taken twenty at a time is ten waves, and a wave is one agent run of several attempts, which the elapsed column below puts in the tens of seconds — hence the four minutes. Raising the concurrency shortens the wall clock and leaves the token cost exactly where it was, so the only real limit on running more is the provider's rate limit. Against a cost like that, a suite running each case once is a suite passing eleven times in twelve on a defect that is already reaching users.

A realistic bad call comes from an ambiguous request, not a corrupted argument#

The candidate call is the part of this test most often got wrong, because the obvious way to produce a bad call is to write one, and hand-written bad calls drift towards the pathological. A status of 42, a null byte, a 10,000-character string and a missing required field all test the validator, which is worth testing and is a different test. No model emits any of them in response to "close this ticket", so a recovery test built on them measures recovery from a situation that will not occur.

The bad calls that occur come from ambiguity between the user's world and the schema's. Four sources produce almost all of them. The interface a human reads renders the value differently from the API — a ticket UI shows "Open" and the enum is open, which is where a casing failure comes from. The instruction names an entity that resolves to more than one identifier, or to an identifier in the wrong namespace, which is the customer id passed where a subscription id was wanted. A natural-language quantity has to be formatted — "next Tuesday" into a date, "£129.99" into minor units — and the model picks a plausible format from several. And a precondition holds in the domain that the schema cannot express, so the arguments are well-formed and the call is still refused: a ticket that must be assigned before it can be closed.

The reliable way to build the corpus is to stop inventing it. Log the arguments of every rejected tool call in production for a fortnight, group them by error code, and take the top twenty; those are the calls your users' phrasings actually produce. Before there is production traffic, the substitute is to sample the model itself: give it the real schema and a realistic instruction, generate the first call fifty times at temperature 1.0, and keep the ones that fail validation. Both methods have the same property, which is that the failing call is generated by the same process that will generate it in production rather than by an engineer imagining what a model might do.

Whether four attempts is a pass depends on what an attempt costs#

An attempt is not a loop iteration. It is a model round trip, a tool invocation against a real service, a chunk of context that stays in the transcript for every subsequent turn, and — on a tool with side effects — an opportunity to commit something. The token cost in particular does not scale linearly with attempts, because each attempt appends the failed call and its error to the context that the next attempt has to re-read.

At $3 per million input tokens and $15 per million output, four attempts costs about $0.038 and nine costs about $0.13, so the ratio is roughly 3.4 rather than the 2.25 that counting attempts suggests. On 40,000 runs a month with 6% reaching the error path, the difference between a four-attempt and a nine-attempt distribution is around $220 a month, which is not the argument. The arguments that matter are the 22 seconds, which is well past the point where a human waiting on a chat response assumes the system has hung, and the nine invocations of a tool that may commit.

All of that comes out of the table and the two prices, and walking it shows where the non-linearity lives. The arithmetic uses the cumulative column rather than the per-attempt one, because each attempt is billed for everything before it as well: 1,850, then 1,850 plus 2,430, and onward, which is why the fourth attempt's cumulative 10,880 is nearly six times the first attempt's input. Multiply 10,880 by $3 per million, add the four output blocks at $15 per million, and the total is about $0.038. Do the same at nine attempts against 38,200 cumulative input and it comes to about $0.13. Dividing the second by the first gives roughly 3.4, while dividing nine attempts by four gives 2.25, and the gap between those two ratios is the compounding — every attempt makes the next one dearer to read. Then 6% of 40,000 runs is a few thousand a month reaching the error path, and that count times the difference between $0.13 and $0.038 is the $220.

The tool invocations are where the accounting stops being about money. Nine attempts against create_refund is nine chances to create a refund, and the only reason it is not nine refunds is the idempotency key doing its work on every one of them. A recovery test on a side-effecting tool therefore carries one more assertion, which is that the store contains exactly one committed effect at the end regardless of how many attempts were made:

assert repo.count_refunds(charge_id="ch_4471") == 1
assert sum(1 for c in trace.calls_to("create_refund") if c.result.get("ok")) >= 1

The budget itself should come from a constraint outside the test. If the product's p95 for a conversational turn is 20 seconds, meaning 95 turns in every 100 come in under that, and the happy path spends 7 of them, the recovery budget is about five attempts and the table above says so directly. The reading is a subtraction and then a scan: 20 seconds less the 7 the happy path already spends leaves thirteen seconds for recovery, and going down the elapsed column, four attempts have consumed 9.6 seconds while a fifth on the same trend lands just inside thirteen and a sixth does not. Five is where the budget runs out, and the number arrived from a product constraint rather than from the suite, which is the whole point of taking it from outside. Choosing the bound by running the test and writing down whatever number currently passes produces a test that ratifies the present behaviour and can never detect a regression in it.

The comparison inspection cannot make#

Running this across three versions of update_ticket — one returning a raw traceback, one returning the exception string, one returning a mapped error object — gives the comparison that reading the code cannot, since all three of them eventually completed the task.

The middle column is the one that pays for the whole exercise. Stripping the frames cut tokens per failing run by 37% and moved the median by one attempt, which looks like progress on a dashboard and left 6.5% of runs failing outright.

The 37% is 14,900 down to 9,400, a fall of five and a half thousand tokens expressed as a share of where it started, and it is a real saving. Set it beside the column on its right, where the same measure falls to 3,100 and the never-recovered rate falls from 8.0% to 0.5%, and the shape of the middle column comes clear: it improved the axis that appears on an invoice and barely touched the axis that appears in a support queue.

Without this measurement the two right-hand columns are indistinguishable, because both of them return a tidy JSON object and both of them mostly work.

When a new model version needs nine attempts#

Six weeks after a mapped version ships, the underlying model is upgraded and the same test needs nine attempts on its 95th percentile, still succeeding, still unattended. The reading that "it recovers, so the criterion is met, and pinning a number makes the suite flaky" treats attempts as noise, which the table above shows they are not: the p95 run goes from about 5 seconds and 4,300 tokens to 22 seconds and 38,000, and on a side-effecting tool from two commit opportunities to nine.

Both of those pairs are read off the same table, and the reason the first pair is the second row rather than the fourth is in the comparison above it: the mapped error object's 95th percentile was two attempts. Two attempts is 4,280 cumulative input tokens and 4.9 seconds elapsed, which is what the quoted figures round to. Nine attempts is 38,200 and 22.1 seconds. So the regression is not one attempt worse, and it is not twice as bad. It is a 95th percentile that has walked from the second row of that table to the last, taking a factor of roughly nine in tokens and a factor of four or five in elapsed time with it, on a criterion that still reports a pass.

Treat it as the regression it is and diff the traces rather than the numbers. The causes are usually specific and fixable. A model that now prefers to call a read tool before correcting is spending two attempts gathering information the error already contained, which means the error is being read past — often because a long message sits below an alternatives array that the model treats as the operative field. A model that repeats a terminal failure is not reading the class, which is a tool-description problem more than an error-body one. A model with a longer default preamble is emitting the same calls with more tokens between them, which changes cost and not attempts. Each of those has a repair on your side of the boundary.

If the new behaviour survives the repairs and is genuinely the model's, raise the budget deliberately, in the same commit, with the new cost and latency figures recorded next to it. What is not defensible is leaving the assertion unbounded, because a criterion phrased as "recovers without human help" and implemented as a loop that terminates measures only that the loop terminates, and it will keep passing while the p95 walks from four to nine to twenty.

The instrument has an honest ceiling. Its numbers are conditional on the bad-call corpus, which is drawn from one model version and one population of users, and it says nothing about failures nobody has emitted yet — the corpus has to be refreshed from production logs on the same cadence as the model changes. It measures recovery from a single malformed call, and multi-step recovery, where the model must undo a committed step before retrying, is a harder problem that this shape of test only approximates. Anthropic's Effective harnesses for long-running agents is the closest thing to guidance on the resumability side of it, and the state of the art there is still mostly engineering judgement.

Recovery is a property of the system you can only observe by running it, and the test that observes it asserts on the trace: an error actually occurred, the store holds the right state, no human intervened, no repair layer sat in front of the validator, and the whole thing took no more than a number of attempts you chose from a latency or cost budget rather than from the current behaviour. Repeat it enough times to see the tail, because a single run of a sampled process is a sample of one drawn from a distribution whose interesting part is at the far end.

What the instrument also supports is an ordering. Point it at the tools you already have, starting with the ones that commit something, and rank them by failure rate multiplied by tokens per failing run: that product is the cost of each tool's error path, in a unit a budget owner recognises, and the tool at the top of that list is the error body worth rewriting first.