Blast radius is partial failures, retry fan-out and lost attribution

The complaint came in through the account team: the policy assistant had told a customer's HR lead that there was no restriction on transferring accrued leave between the UK and German entities, and there is one, and it is three paragraphs long in a document the assistant has indexed. The run looked healthy. No errors surfaced to the user, latency was 5.1 seconds, and the answer was fluent and cited two documents.

What the trace showed was that the orchestrator had split the question into five sub-questions and dispatched five retrieval workers. Four returned. The fifth — the one asking about cross-entity transfer rules — hit a 4-second timeout on the search backend, returned an empty result set, and the aggregation prompt received four filled slots and one empty one. Nothing in that prompt said what to do with an empty slot, so the model did the natural thing with an absence of evidence and wrote that no restriction was found.

"Operational blast radius" is the phrase people use for this, and as a phrase it loses arguments, because it sounds like a way of saying you prefer simple things. The phrase is also a borrowed picture, and it is worth saying where the picture stops holding: a real blast radius is a distance, and damage falls off as you move away from the centre. What follows does not fall off. It is a fixed set of obligations that are owed on every request, in the quiet weeks as much as during an incident, which is why converting the phrase into a list of specific obligations is worth doing — that list is what the design is charged every day it serves traffic, and it is what a one-point quality gain is being asked to pay for.

What one worker failing does to the arithmetic#

The first obligation is a written answer to the question "what does aggregation do when a worker fails or returns nothing?" — and it has to be written, because the default is that a language model interprets an empty slot as evidence of absence.

That default is worth deriving rather than asserting, because it is the step where an engineer's intuition about failures and a model's behaviour part company. The aggregation prompt is assembled by pasting each worker's result into a slot, so an empty result contributes no text. What arrives at the model is therefore a prompt containing four pieces of evidence and nothing where the fifth would be. Nothing in that prompt distinguishes "we searched for cross-entity transfer rules and the corpus contains none" from "the search timed out and nobody looked", because both produce the same absence of characters. Asked to answer from the evidence given, the model writes the answer the evidence supports, and the evidence given contains no restriction. The confident wrong answer is not the model ignoring an error; it is the model never being told there was one.

The reason this comes up more often than teams expect is multiplication. A search backend with 99.2% success under load is unremarkable and would never page anyone if you called it once per request. Fan it out five ways and the probability that every worker succeeds is 0.992⁵, which is 96.1%, so roughly one request in 26 has a hole in it.

The power in that expression is doing all the work, and it comes from one assumption stated plainly: if each call succeeds independently of the others, the chance that all five succeed is the chance the first one does, times the chance the second one does, and so on — one factor per call, which is what raising the number to the fifth power means. Subtracting from one gives the share of requests with at least one hole, a little under four per cent, and turning a share into "one request in how many" is dividing one by it, which lands at twenty-six.

Read the first four rows as gaps rather than as levels: holding the backend fixed and moving only the width, the hole rate roughly doubles from three workers to eight. The last row moves a different quantity and is the encouraging one, and it is also the expensive one, since taking a dependency from three nines to four is usually a quarter of somebody's roadmap.

The residual on that arithmetic matters more than the arithmetic. Independence is the assumption that one worker failing tells you nothing about whether another will, and the situations that produce holes are exactly the situations where it is false: a backend under load fails several concurrent requests together, so failures arrive in clumps rather than sprinkled evenly. Correlated failure moves the true figure in the bad direction — more requests with holes, and holes concentrated in the minutes when the most people are asking. The table is therefore a floor on the problem rather than an estimate of it.

The cheap fix is to make the hole explicit rather than to make holes rare:

def aggregate(subtasks: list[SubtaskResult]) -> Answer:
    holes = [s for s in subtasks if s.status != "ok" or not s.evidence]
    if any(s.required for s in holes):
        raise PartialFailure(missing=[s.id for s in holes])   # fail the run
    slots = "\n".join(
        f"[{s.id}] {s.evidence}" if s not in holes
        else f"[{s.id}] NO EVIDENCE RETRIEVED — status={s.status}"
        for s in subtasks
    )
    return model.complete(
        system=AGGREGATION_PROMPT,   # "an unretrieved slot is unknown, not absent"
        user=f"{question}\n\n{slots}",
    ).with_metadata(degraded=bool(holes), missing=[s.id for s in holes])

The line that repairs the failure is the one putting text where the absence used to be. Once the slot reads NO EVIDENCE RETRIEVED, the two situations the model could not tell apart have different prompts, and the instruction in the system message about an unretrieved slot being unknown has something to attach to. Three decisions are encoded there and each one had to be made by a person: which subtasks are required rather than optional, what a hole looks like in the aggregation prompt, and whether a degraded answer is allowed to reach the user at all. A single loop never faces them, because when its search times out the model sees the timeout in the tool result and can retry, reformulate, or say it does not know, inside the same context that produced the query.

Retries multiply, and they multiply worst when it hurts#

The second obligation is that retry policy stops being local. A single loop that retries a failed tool call twice makes at most two extra calls per request. A five-way fan-out with the same policy makes up to ten extra calls, and it makes them concurrently, against a backend that is failing precisely because it is under pressure. The ten is the same multiplication as before: a per-branch budget of two, applied to five branches that all exist at once, where the loop has one branch by construction and so its budget is its total.

Steady state for the policy assistant's orchestrator is 7 model calls per request: one plan, five workers, one aggregation. During a 20-minute period when the search backend's error rate rose to 20%, the mean rose to 9.4 calls and the p99 to 21, because a bad request is one where several workers each burn their retries. Load on the search backend during that window was 2.6× the pre-incident baseline, generated entirely by the retry policy, which means the fan-out amplified the brownout it was reacting to. A single loop degrades differently: it makes one call at a time, so a failing backend slows the run rather than multiplying it, and the natural pressure valve is the iteration cap.

The third obligation is recoverability. Deploys, pod evictions and provider hiccups all kill runs mid-flight, and for a single loop the recovery story is to retry the request with the same idempotency key and pay for it twice. An idempotency key is a value the caller supplies with a request so that a second request carrying the same value is recognised as the same request rather than as a new one, which is what makes a blind retry safe: the work either happened once or happens once now, and the caller cannot tell which. That mechanism works for a single loop because a loop's unit of work is the whole request, so there is exactly one thing to deduplicate. For a coordinated run, three workers may have finished and two may not, and there is now a real choice: persist per-subtask results keyed by (run_id, subtask_id) so a resumed run only redoes the missing ones, or redo everything and eat the cost. Persisting is the better answer and it is a piece of infrastructure — a store, a schema, an expiry policy, and a resume path that has to be tested by killing runs on purpose. This is the part of the bill most often left off the estimate, and it is where the durable-execution engines earn their keep — frameworks that record each completed step to storage as it finishes, so that a run killed halfway through resumes from the last recorded step rather than starting again; adopting one converts an unbounded amount of bespoke state handling into a known dependency with its own operational surface, which is a trade rather than an escape.

Caps have to hand something back, and attribution gets harder#

The fourth obligation is that every cap needs a defined return value. A loop capped at twelve iterations that simply stops has produced an undefined result, and callers will treat undefined results as answers unless the contract forbids it. In a coordinated design the caps compose: an orchestrator allowed six planning rounds over workers each allowed twelve iterations has a worst case of 72 model calls per request, and that number is a product rather than a sum, since each of the six rounds may dispatch work that runs the full twelve. Knowing it matters because it is what a pathological input costs you. The contract worth writing is that a capped run returns whatever partial result exists plus a machine-readable reason — {"status": "capped", "completed_subtasks": 3, "of": 5} — so that the layer above can decide between showing a degraded answer, falling back to a simpler path, or queueing for a human.

The fifth is attribution, and it is the one that compounds over months rather than during an incident. When three calls produce one answer, no single trace tells you which call was wrong. The wrong answer about leave transfer was produced by a healthy aggregation step working on a bad input, so a grader looking only at final outputs records it as an aggregation failure, and the prompt engineer spends a week improving an aggregation prompt that was never at fault. Getting attribution back requires grading the intermediate artefacts: was the plan's decomposition correct, did each worker retrieve what it should have, did aggregation preserve what it received. That is a second evaluation layer, with its own labelled data, its own maintenance and its own drift.

The size of that second layer follows from what it has to answer. Each of those three questions is a different judgement about a different artefact, so each needs its own labels: a set of questions paired with the decomposition a competent person would have written, a set of sub-questions paired with the passages that should have come back, and a set of worker outputs paired with what a faithful merge of them looks like. None of the three can be derived from the end-to-end labels you already have, because the end-to-end label records whether the final answer was right and says nothing about which intermediate produced it — which is the original problem restated. And each set has to be maintained as the prompts change underneath it, since a decomposition labelled against last quarter's planner may not be the decomposition this quarter's planner should produce.

The left column is where the argument usually gets lost, because four of its six entries describe an absence rather than a cheaper version of the same thing. "Not possible" is not a policy that happens to be simple; it is the observation that a design with one branch has no partial state for a policy to be about. That is the sense in which these are obligations of the shape and not of the team: they arrive with the second branch, whoever writes it.

What the good version of this costs#

None of these obligations is unsolvable, and it is worth being specific about what solving them takes, because "just use a durable execution engine" is both true and a larger commitment than it sounds. A workflow engine or a graph framework with checkpointing gives you the resume path, per-step retry policy with backoff, and a run history you can inspect, which covers the third obligation outright and most of the second. In exchange you take on a new deployment, a versioning problem when a workflow definition changes while runs are in flight, and a debugging story where the stack you read is the engine's rather than yours. The versioning problem is worth seeing concretely, because it is the one that surprises people who have only used such an engine on short jobs. A resumed run replays the steps already recorded and then continues into the definition the process is running now, so a run that started under one version of the workflow can finish under another. If the change added a step, the resumed run skips it; if it renamed one, the recorded history no longer lines up with the code; if it altered what a step returns, the next step receives a shape that the version it was written against never produced. Every engine has an answer to this, and every answer amounts to keeping old definitions alive and deciding which runs belong to which, which is a piece of ongoing work rather than a setting. For a team already running one for other reasons, that is close to free. For a team adopting one solely to make a five-way fan-out recoverable, it is a quarter of platform work bought with the one point of quality from chapter 4.

The partial-result and attribution obligations are not solved by any engine, because they are semantic. An engine can tell you that a step failed; it cannot tell you whether the answer was still worth sending. Only a person can say whether the cross-entity transfer sub-question is required or optional, and only a labelled set of plans and worker outputs can tell you whether a bad answer came from a bad decomposition. Budget those as ongoing work rather than as a one-off.

None of this is an argument against fan-out. The market-scan feature that researches eleven named vendors inside a 30-second budget has no other option, and its team pays every line in that table knowingly, which is the correct way to pay them. The single loop is not exempt from operational cost either — it has context growth, cap hits, and its own tendency to repeat a failing tool call — and those show up in different rows of the same kind of table.

What the itemisation does is make the trade quotable. When a proposal offers one point of quality for 2.4× the model calls, the counter-argument is no longer that coordination feels heavy; it is six specific obligations, four of which require code that does not exist yet. Chapter 6 puts that against the quality number and makes the call, including the awkward case where the design in question is already serving traffic.