Recovery is measurable only if you break something on purpose

Recoverability is what an agent does when a tool it depends on fails: whether it reaches the goal by another route, stops in a state a person can act on, or carries on as though nothing had happened. What it costs to have no measurement of it is easiest to see from an incident. At 09:14 on a Tuesday the calendar provider started returning 503 on roughly one call in three, and kept doing it for twelve minutes. Three hundred and forty agent runs were in flight or started during that window. Sixty-one of them created duplicate events when the API recovered, because the agent had read the timeout as a failure, retried calendar.create_event, and the original write had in fact succeeded. Nineteen ran until they hit the loop guard at fifty steps — the hard ceiling on iterations that the harness enforces so that a stuck run stops instead of continuing indefinitely — having cycled between availability lookups that returned partial data. Four told the user, correctly, that the calendar was unavailable and they should try again shortly.

The eval set had two hundred tasks and had reported 94% success the previous week. It could not have predicted any of this, because every fixture in it returned a well-formed 200. The recoverability column in the trajectory report from the last chapter — the per-run record of every tool call an agent made, in order, with its arguments and its result — was empty on all two hundred rows, and it was empty for a structural reason: recovery is a response to a failure, and there had never been one.

An unmodified run contains no recovery to measure#

This is the one metric in the topic that cannot be extracted from a run you observe. Success, step counts, unsafe actions and cost are all readings taken off an unmodified execution. Recoverability will not come off a run that way, and the chain behind that is short enough to check link by link.

Recovery is a response to a failure and has no other content: an agent that reaches the goal on its first attempt has not recovered from anything, it has succeeded. A response cannot be observed where there was nothing to respond to, so a run in which no tool call failed holds no moment at which the agent's handling of a failure could have shown itself, and no amount of care in the scoring conjures one. Every fixture in the eval set returns a well-formed success, so not one of its two hundred tasks contains a failure for an agent to respond to. The column was therefore blank on every row because of how the set was built rather than because the agent never recovers, and a larger set built the same way would be exactly as blank.

That leaves one move. If the column is blank because no fixture ever fails, the only way to get a number into it is for a fixture to fail, which means the harness has to hand the agent something the real tool would not have returned on that call. Recoverability requires you to change the execution, which means the eval harness has to be able to lie to the agent.

Injecting faults deliberately rather than waiting for them#

The practice has a lineage worth knowing, because it explains what is different here. Netflix's Chaos Monkey, from 2011, killed production instances at random to force teams to build systems that tolerated instance loss; Jepsen did something similar for distributed databases by partitioning networks under load and checking the consistency claims that survived. Both inject faults randomly into a live or realistic system and look for violations of an invariant.

An agent eval wants the opposite of randomness. The point is a repeatable number that can be compared between releases, so the fault has to be deterministic: this task, this tool, this step, this fault, every time. That makes it closer to a unit test with a mocked failure than to chaos engineering, with the important difference that the thing under test is a model's policy rather than a code path, and no amount of reading the source tells you what it will do.

That lineage is an analogy here rather than a description, and it breaks in two places worth naming. The first is the randomness, which in chaos engineering is where the value comes from: the invariants are written down in code, so a state nobody imagined still produces a verdict automatically, and the unimagined states are what you are paying for. A fixture cannot reach a state nobody imagined, because somebody had to write the fixture. The second break is in what a green result establishes. A code path is deterministic given its inputs, so watching it survive a fault once settles how it handles that fault; a model's policy is not, and the same task with the same injected fault can produce a different continuation on a different run. One faulted run is therefore an observation of what happened once rather than a proof of what the agent does.

The catalogue that covers most of what production actually does is short.

The teaching in that table is in the middle column, and it is a difference between groups of rows rather than anything about a row on its own. The top three rows present as errors: something raises or comes back with a status attached, and every layer between the tool and the model can see the call went wrong. The bottom four present as successes — a valid response with zero rows, a body with a field missing, a read showing state from before the write, a send that delivered some of its invites — because in all four the transport worked and the call returned. A fault in the first group is caught by whatever handles errors; a fault in the second can only be caught by whatever is deciding what to do next, which is the model.

The two that teams skip are the two that produce the worst production incidents. An empty result with a 200 is not an error by any technical definition, so nothing in the stack raises, and the agent has to notice semantically that it got nothing. A stale read after a write is what turned twelve minutes of 503s into sixty-one duplicate meetings.

The injection policy is part of the eval set#

Faults are attached to tasks by a written policy, kept in the same file as the tasks, because the alternative — injecting randomly at run time — makes two releases incomparable and makes a failing run impossible to reproduce.

FAULT_PLAN = [
    # task_id, tool, at_call_index, fault, mode
    ("sched-014", "list_availability",     1, Empty(),            "single"),
    ("sched-014", "list_availability",     1, Empty(),            "sustained_20s"),
    ("sched-027", "calendar.create_event", 0, Timeout(30),        "single"),
    ("sched-027", "calendar.create_event", 0, TimeoutButApplied(),"single"),
    ("sched-031", "mail.send_invite",      0, Partial(ok=3, rejected=2), "single"),
    ("sched-045", "list_availability",     0, Http(503),          "sustained_12s"),
    ("sched-045", "list_availability",     0, Http(429, retry_after=20), "single"),
]

class FaultingToolClient:
    def __init__(self, inner, plan, task_id):
        self.inner, self.task_id = inner, task_id
        self.plan = [p for p in plan if p[0] == task_id]
        self.counts = defaultdict(int)

    def call(self, name, **kwargs):
        n = self.counts[name]
        self.counts[name] += 1
        for _, tool, idx, fault, mode in self.plan:
            if tool == name and self._active(idx, n, mode):
                return fault.apply(self.inner, name, kwargs)   # may still perform the write
        return self.inner.call(name, **kwargs)

TimeoutButApplied is the fixture that earns its keep. It performs the underlying write, then raises a timeout at the client, which is precisely the state the provider left agents in during the incident. An agent that retries blindly produces two events and fails the single_event assertion from chapter 1 — the end-state check that the finished task left exactly one calendar event behind; an agent that reads back before writing again produces one. No fixture that merely returns an error can tell those two agents apart.

The catalogue's timeout row and that fixture look like two different prescriptions and are the same one. The row says to retry once, then treat the state as unknown and verify before writing again, and verifying before writing again is exactly what TimeoutButApplied discriminates on. A plain Timeout(30) leaves the verifying agent and the blindly retrying one both holding a single correct event, so the row states a policy that only this fixture can test.

The mode field separates a single-shot fault from a sustained one. Single-shot answers "does it retry"; sustained answers "does it know when to stop", and sustained is where loop behaviour shows up. On the scheduling set, sustained faults were where the median step count went from 6 to 23, and where two tasks hit the loop guard.

The injection point decides whose policy is under test#

One question the plan has to answer before any of this is useful is who is supposed to handle the fault. Most tool clients already retry transient errors internally with backoff, which means a fault injected below that layer never reaches the model at all and what you measure is the retry policy in your HTTP client. That is a legitimate thing to test and it is not recoverability of the agent.

Injecting above the client — at the boundary where the tool result is handed back to the model — measures the agent's policy, and the two injection points give genuinely different numbers. On the scheduling agent, injecting a single 503 below the client produced a 100% recovery rate, because tenacity, the Python retry library wrapping the HTTP client, retried it and the model never knew; injecting the same 503 above the client dropped recovery to 71%. Both numbers are true and only one of them is about the agent, so the plan records the injection point alongside the fault.

The below-client number is the dangerous one, and it is dangerous in a particular shape: it stays where it is while the property it appears to be guarding moves freely underneath it. tenacity retries a 503 regardless of what the model would have done next. Change the system prompt so the agent stops checking results for errors, swap in a model that abandons a task at the first sign of trouble, delete the retry guidance from the tool descriptions entirely — the below-client recovery rate does not move, because the model was never in the loop that produced it. A quantity that cannot fall when the thing it is supposed to protect gets worse is not a measurement of that thing, whatever the column is headed.

Believing that number is tempting for reasons that are not silly. It is a real measurement of a real defence: the retry library genuinely does absorb transient errors in production, and the runs behind the number genuinely did finish the task. It is also green, and a green figure under the word recoverability is very easy to file as recoverability. The error is not in the measurement, it is in what the measurement is taken to be about — a fact about an HTTP client read as a fact about the agent's policy, and the two coincide only while the client happens to be handling everything.

It is the gap between the two figures that carries the finding rather than either level: the higher one is how much the stack absorbs before the model is involved, the lower one how much survives when the model has to deal with the fault itself. The constancy is also specific rather than general, since it holds for exactly the fault classes the client is configured to retry. An empty result set with a 200 status is not one of them, so injecting it below the client passes it to the model unchanged — which is another way of saying that the faults a retry library can hide from your eval are the ones it was already handling in production.

Three outcomes, and why collapsing them loses the point#

A faulted run does not divide into pass and fail in any useful way. There are three distinguishable good-looking results and they carry different information.

The first is genuine recovery: the agent observed the fault, changed what it was doing, and reached an end state that passes the chapter 1 assertions. The step budget for a faulted run is relaxed — a recovery that costs four extra calls is what recovery looks like — but it is still a budget, and a run that recovers in thirty-one calls against a base budget of eight is reported as recovered and over budget at once.

Relaxing that budget is a decision, and the requirement underneath it is narrower. What is forced is only that a faulted run cannot be scored against the base budget, since noticing a fault and doing something about it costs calls the clean run never had to make, so that budget would report every genuine recovery as over budget. What is chosen is how much slack to allow, and every answer to that is a judgement about how much extra work a recovery is worth paying for. Unlimited slack is a defensible choice with a cost attached: a run that thrashes until the loop guard and lands on the goal anyway would then be reported as a clean recovery.

The second is a clean abort: the agent did not reach the goal, said so specifically enough that a person could act on it, and left nothing half-done behind. Scoring this needs both halves. "I wasn't able to complete that" with a calendar hold still sitting on three people's Tuesdays is not a clean abort, so the classifier checks the effectful spans that ran after the fault and requires the world to be back where it started or explicitly described.

The message itself is the part that is easy to score badly. "Something went wrong, please try again" is technically an abort and is worthless to the person reading it, because it does not say what state their calendar is in. A useful check requires the final message to name what was attempted, what state it ended in, and whether any action is needed — three facts a judge can verify against the span record with a narrow prompt, since it only has to confirm that stated facts match recorded ones rather than assess tone. Runs whose message asserts something the trace contradicts, such as claiming nothing was created when a hold exists, are worse than a bare failure and should be classified as failures rather than clean aborts.

The third is the one that gets lost when teams score faulted runs as pass or fail. An agent can pass every assertion without ever registering that anything went wrong. Task sched-014 injects an empty availability result; the fixture calendar for that task happens to be genuinely free on the requested afternoon; the agent, receiving zero rows, books 14:00 anyway and lands a correct meeting. The end state is right, the trajectory is short, the run scores clean, and the policy it demonstrates is that an empty availability response leads to booking on an assumption. Run that against a busy calendar in production and it double-books an executive.

def classify_faulted_run(spans, verdict, fault) -> str:
    fault_span = next((s for s in spans if s.step == fault.step and s.faulted), None)
    observed = fault_span is not None and (
        fault_span.error is not None or fault_span.result_rows == 0)
    after = [s for s in spans if s.index > fault_span.index] if fault_span else []
    changed_strategy = any(s.name != fault_span.name for s in after if s.kind == "tool_call")
    residue = [s for s in after if TOOL_EFFECTS.get(s.name, {}).get("effect") != "read"]

    if verdict.score == "pass" and observed and changed_strategy:
        return "recovered"
    if verdict.score == "pass" and not changed_strategy:
        return "unnoticed_success"          # right answer, wrong policy
    if verdict.score != "pass" and not residue and spans[-1].kind == "final_message":
        return "clean_abort"
    return "failed"

unnoticed_success is the column that changes conversations, because it is the only one where the outcome metric and the safety judgement point in opposite directions. It is also the label a pass/fail scheme has no room for, since the run it describes is a pass by every end-state check the harness applies.

The four labels are worth reading as a set rather than one at a time. Every path through classify_faulted_run returns a string, those four strings are the only ones it can return, and each run gets exactly one of them. The four counts over a faulted set therefore add up to the size of that set exactly. That exhaustiveness licenses reading the counts against one another as a decomposition: if clean aborts rise between two releases, some other bucket fell by the same amount, and asking which one is a well-posed question with an answer in the data. Four tallies that did not partition the population would support no such question, because a rise in one could mean anything at all.

What the identity does not license is treating any of the four counts as a rate that describes production. The denominator is the faulted set, and the faulted set is the faults somebody chose to inject on the tasks somebody chose to inject them into. Add a handful of stale-read fixtures to the plan and the recovered share moves, with no change to the agent, the model or the prompt. The counts are a property of the injection plan first and of the agent second, and they describe production only to the degree that the plan resembles it.

That residual is the argument for a decision, and it is worth labelling as a decision rather than as something forced. The faulted runs are kept out of the headline success rate, and that separation matters more than it looks. A faulted run is a different question from a clean one, and averaging them together produces a number that moves when you add fault fixtures, which makes it impossible to compare against last month. Nothing in the harness requires the separation, and a team wanting one figure could compute one — it would inherit the plan-dependence above and stop being comparable release to release, which is the only property a headline rate exists to have. The report carries a task success rate over the unfaulted set and a resilience breakdown over the faulted set, and the two are read side by side.

Where injected faults stop being informative#

The faults in your plan are the ones somebody thought of, and the distribution of real failures is not the distribution of imagined ones. That is the same limit the chaos-engineering comparison ran into, reached from the other side: random faults against a live system arrive at states nobody enumerated, and a fixture set can only contain what was enumerated. The twelve-minute 503 window was correlated across every call in a run and across every run in the window, which is a very different stimulus from one injected 503 on one call. Sustained modes narrow that gap and do not close it, because production failures also arrive with elevated latency on the calls that do succeed, with retries from other services adding load, and occasionally with a second unrelated fault on top.

There is also a limit on what a fixture can simulate. Injecting a TimeoutButApplied requires the harness to know that the write applied, which is exactly what the agent cannot know; that asymmetry is the reason the fixture is useful and also the reason it is easy to build a fixture that is unfair — one that punishes an agent for a decision that no available information could have made correctly. When a faulted task starts failing every candidate agent, the first thing to question is whether the task is answerable at all.

Recoverability is now a column with numbers in it, which completes the trajectory report begun in the last chapter. Those numbers exist because the harness was given permission to return something the tool would not have, and they mean what they mean only inside the plan that decided which lies to tell where. Filling the column also surfaced something the report cannot explain: in six of the nine unnoticed successes, the agent's response to the empty availability result was to call list_availability again with identical arguments, and in the other three it called nothing at all and went straight to booking.

Whether calling that tool, with those arguments, at that moment, was the right action is a question about tool choice, and it needs a measurement that looks at single decisions rather than whole paths. None of the four labels can settle it, because all four are verdicts on a whole run: a label says where a trajectory ended up, and this question is about one step inside it.