Reversibility and blast radius decide what a wrong action costs

An accuracy figure is a property of a component, and a component does not have consequences. What has consequences is the action the figure is attached to. Reversibility here means whether executing an undo restores the state that existed before, judged in the world rather than in the code. Blast radius means how far one wrong action reaches — a single record, a whole table, or a party outside the company. Neither appears in an evaluation report, and both change the design more than the evaluation does.

Without them, a design review compares systems on their pass rates and treats the result as the answer. That works while both candidates do the same thing with their conclusions. It stops working the moment one of them writes, pays or sends, because the same figure then describes two situations whose worst cases are separated by several orders of magnitude of cost and by whether anyone can put things back.

The same error rate, attached to two different endings#

Take a rate of four percent and attach it to two systems that read the same receipts. A read-only expense summariser producing a wrong category 4% of the time generates a mildly inaccurate report that somebody notices at quarter end. The same 4% attached to auto-approval is roughly 744 wrong payments a year.

That figure is a product with nothing left over, and it is worth building rather than quoting. Sixty-two percent of one company's expense claims fall under fifty dollars, and that share amounts to about 18,600 claims a year. Two instruments meet there: the share and the count together pin the total volume at thirty thousand claims a year, which is the figure the same system reports independently when it counts everything it processes. Because the two arrive by different routes and agree, the volume is not an assumption anyone has to defend. Multiply the auto-approved subset by the error rate — eighteen thousand six hundred by four in a hundred — and the answer is the 744. Nothing else enters the calculation, so no further cause hides inside that number; it is the error rate and the volume, and moving either moves it proportionally.

The cost is not primarily the money. At an average of $31 that is $23,000, which a company processing 30,000 claims will absorb. The cost is that 744 transactions entered the accounting system through a control that cannot be described to an auditor, and unwinding one of them requires a payroll adjustment, an HR conversation with the employee, a ledger correction and a VAT amendment.

Each one that clears the gate becomes a line on the next payroll run. Two weeks later the money is in someone's bank account, the general ledger has a coded entry, the VAT reclaim has been filed on it, and the claim appears in a quarterly report that the finance director has signed. Getting one of those wrong is not a bad output. It is a payment, and the difference matters more than any accuracy number, because the same 4% error rate means something entirely different depending on what the system does with its conclusion.

Reversibility is a property of the world, not of the code#

The instinct from distributed systems is useful here and needs one correction. Sagas taught everyone to think in compensating actions: for every operation, write the operation that undoes it, and treat a workflow as reversible if every step has one.

That is a borrowed picture and it is worth saying so before leaning on it, because it holds exactly where it was built to hold. A saga's compensating action works on state the saga owns — a reservation it made, a row it wrote, a counter it moved — and running the compensation returns that state to what it was. The correction is that a compensating action is only a reversal if executing it restores the prior state, and for actions that touch money, people or third parties it usually does not. A refund is a second transaction, not the absence of the first. The employee has been told they were paid and is now being told they were not; the ledger has two entries where it should have had none; the tax filing was made on a figure that has changed. The picture breaks at the boundary of what your system owns, and every item in that list is outside it.

The belief the picture produces is the one to name: that because a mechanism to undo exists, the action is reversible. It is tempting because the mechanism is real and can be demonstrated in a test. What varies underneath the constant mechanism is everything that decides what the undo costs, and none of it is visible from the code that performs it.

So reversibility is best assessed with three concrete questions rather than a yes or no. Who can undo it — the system automatically, an operator with a button, or a person in another department with a ticket queue? Within what window — before the nightly batch, before the payroll run, before the quarter closes, never? And what does the undo itself cost, counting the parts that are not engineering work? Applied to the expense flow, the auto-approval is undoable by finance, within the window until payroll runs on the 25th, at a cost of roughly two hours across three people and one uncomfortable email. That places it firmly on the irreversible side for design purposes, even though a mechanism exists.

Those three questions are what the single word hides. A yes-or-no field on a design document records that a compensating action was written; the questions record who is on the hook, how long they have, and what it takes out of them. Two systems can both answer yes and sit on opposite sides of the grid below.

How far one wrong action reaches#

Blast radius is the second axis and is not the same as volume. It asks how far one wrong action reaches: one record, one user's data, one team's dashboard, every row in a table, or something outside the company entirely. A single wrong expense category is one record. A model that decides to re-categorise every claim from a supplier because it inferred a pattern touches thousands. A model that emails the supplier to query a receipt has crossed a trust boundary — the line past which your own controls stop applying, because the action has reached a party you do not administer — and no internal control can pull that back.

That grid decides where a gate goes, and it applies identically to a workflow and to an agent. The useful reading is not the four cells but the distance a single wrong decision can travel across them. In the fixed graph, a wrong categorisation stays in the top-left cell, because the only path from categorise to pay runs through the amount comparison, which is code. In a loop where the model has a payment tool, a wrong categorisation and a wrong payment are the same kind of event — a tool call — separated only by whatever the prompt says.

The step there is worth making explicit. In the graph, moving from a category to a payment requires traversing an edge, and that edge carries a condition somebody wrote. A wrong category arriving at the comparison is still a wrong category; the comparison reads an amount, not a category, so the mistake cannot propagate through it. In the loop there is no edge to traverse. Both actions are entries in the same tool list, reached by the same mechanism, and the only thing standing between a mistake in one and a mistake in the other is text in a prompt.

Every transition in that diagram is an edge in code with a condition you can read. AwaitingManager is a durable state with a row in a table, an assignee and a timestamp, which is what makes an approval gate a gate rather than a suggestion. A state that exists only as a pause inside a running function is not a gate, because nothing survives a restart and nobody can be assigned to it.

The threshold on that edge is a design decision and not a requirement. The requirement is that claims get reimbursed and that the books can be audited; the fifty-dollar line, the payroll date it has to beat, and the fields the approver is shown are choices somebody made and can revisit. Keeping the two apart matters when the design is challenged, because a requirement has to be met and a decision has to be justified, and the justification for this one is given further down.

What the gate has to put in front of the approver#

A gate that shows the model's recommendation and an Approve button is a rubber stamp with an audit trail, and it will be used as one. The measurable symptom is the override rate: if managers approve 99.6% of what reaches them, the gate is not exercising judgment, it is laundering the model's judgment into a human signature. The payload has to contain what an independent decision needs:

@dataclass(frozen=True)
class ApprovalRequest:
    claim_id: str
    amount: Money                      # 84.20 GBP
    receipt_image_url: str             # the source, not a description of it
    extracted_fields: dict[str, str]   # merchant, date, VAT, line items
    extraction_confidence: dict[str, float]
    policy_clauses_matched: list[Clause]  # quoted text + document version, e.g. "T&E v4.2 §3.1"
    policy_verdict: Literal["within", "outside", "unclear"]
    reason: str                        # why it is here: over threshold / unclear / duplicate suspected
    claimant_history: HistorySummary   # 14 claims this quarter, 1 previously rejected
    similar_claims: list[ClaimRef]     # same merchant, same week

Two of those fields are the ones teams leave out. receipt_image_url gives the approver the original rather than the system's reading of it, so a mis-extracted total is catchable. policy_clauses_matched quotes the policy text and pins the document version, so the approver is checking against a rule rather than against an assertion that a rule was checked.

Both omissions have the same shape, which is why they go together. Without the image, the approver can only check the system's reading against the system's reading. Without the quoted clause, they can only check that a policy check happened, not what it concluded. In each case the material needed to disagree is missing, and an approver who cannot disagree will approve — which is the mechanism behind a rubber-stamp rate, rather than inattention on anyone's part.

Why the record is free in one design and reconstructed in the other#

Auditability is usually raised as a separate requirement and it falls out of the same construction. In the fixed graph, the record of what was decided is a byproduct of execution: each transition writes a row naming the from-state, the to-state, the condition that fired, the inputs it read and the policy version it read them against. Answering "why was claim 8814 auto-approved on 3 March" is a query, and it returns the same answer next year because the rows are facts.

The chain under that is short and every link is checkable. An edge fires only when a named condition is evaluated over named values, so both the name and the values exist at that moment. Writing them down costs one insert on a path that was already doing a database write. Because the row records the condition rather than a narrative about it, answering the question later requires no interpretation. And because the row is immutable and carries the policy version in force at the time, the answer does not move when the policy or the model does.

The row is worth seeing, because its columns are what an auditor asks for and they are all available at the moment the edge fires:

CREATE TABLE claim_transitions (
  claim_id        text        not null,
  seq             int         not null,
  from_state      text        not null,   -- 'PolicyChecked'
  to_state        text        not null,   -- 'AutoApproved'
  rule            text        not null,   -- 'amount_lt_threshold'
  rule_inputs     jsonb       not null,   -- {"amount":"31.40 GBP","threshold":"50.00 GBP"}
  policy_version  text        not null,   -- 'T&E v4.2'
  model_versions  jsonb       not null,   -- {"extract":"...-20260115","categorise":"...-20260115"}
  actor           text        not null,   -- 'system' | 'manager:e.reyes'
  at              timestamptz not null,
  primary key (claim_id, seq)
);

rule and rule_inputs are the columns that only a fixed graph can populate honestly, because they require that a named condition was evaluated over named values. policy_version is what makes the row answerable years later, when the policy has been revised four times and the question is whether the decision was correct under the rules in force that day. Storage is about 400 bytes per transition and five transitions per claim, which is 60 MB a year at this volume — a rounding error against seven-year retention.

A model-chosen trajectory does not produce that record; it produces a transcript, and the answer to the same question has to be reconstructed from it. The transcript shows the tool calls in order and the text the model emitted, which tells you what happened and leaves the decisive fact — that this claim was under the threshold and the threshold was fifty dollars — as something inferred from prose. That reconstruction is possible and it degrades in three specific ways: the prompt that produced the trajectory has since changed, the model version has changed, and the transcript is 40 KB per claim, which at 30,000 claims a year is a retention decision somebody has to fund for the seven years that finance records are kept.

The size comparison is worth doing because it runs the opposite way to intuition. Four hundred bytes across five transitions is about two kilobytes a claim, against forty for a transcript — twenty times the storage, for a record that answers the auditor's question less directly and stops being interpretable when the prompt changes. Paying twenty times as much for a worse answer is the part that surprises people, and it is a consequence of storing a narrative instead of a decision.

This is the argument that beats a workflow is cheaper to build, and it is worth being blunt about why the cheaper-to-build framing loses. Build cost is a one-time number and it is the weakest thing on the table, because the moment someone answers that the agent is more future-proof, the comparison becomes six weeks against an unbounded future benefit and the workflow loses. Reversibility and blast radius are not one-time costs. They describe what every wrong action does, forever, at whatever volume the system runs at, and they are the reason the control belongs in code regardless of what the build estimate says.

Where the axes mislead#

They can be used to justify gating everything, which produces approval fatigue and an override rate that trends to zero — a control that exists on paper and not in practice. A gate is worth having only where the approver is given the material to disagree and where the rate of disagreement is measured.

The override rate has an interpretable range in both directions, which makes it a better health metric than most. Below about 2%, the approvers are not exercising judgment and the gate is adding latency and a signature without adding a control; the remedy is either to raise the auto-approval threshold, so that fewer claims reach a human, or to fix the payload so that disagreement is possible. Above roughly 25%, the extraction or the policy check upstream is wrong often enough that the humans are doing the work rather than checking it, and the remedy is upstream. The 6.1% observed here sits in the range where the gate is doing what gates do. None of those thresholds are laws, and what makes them usable is that the number is collected at all — a gate with no override metric is an assertion that a control exists.

Both ends are diagnostic for the same reason, which is that the rate measures the gate rather than the claims. A gate exists to catch decisions the machine got wrong, so a rate near the floor means either that nothing is being caught or that nothing can be, and a rate near the ceiling means the humans have become the extraction step. Only the band between them describes a gate that is checking work which is mostly right.

Time is the other thing a gate spends. AwaitingManager is a durable state, which means claims sit in it, and the median time to approval on this flow is 19 hours with a p95 of 4.2 days. That is a product cost paid by employees waiting for reimbursement, and it is the reason the threshold is set at fifty dollars rather than at zero: gating everything would move 30,000 claims a year into a queue whose throughput is bounded by managers reading things, and the resulting delay would be a worse outcome than the 744 wrong small payments.

They also say nothing about frequency. A large, irreversible blast radius occurring once every three years may be correctly handled by a runbook rather than by a mechanism, while a small irreversible action happening 18,600 times a year deserves a gate in code. Multiplying the two axes by volume is the third dimension, and it is the one that turns this grid into an actual recommendation.

One quantity in all of this is named and never multiplied out, and it should be said plainly rather than left for a reader to notice. The money was priced: the wrong payments, at their average value, come to a sum the company absorbs. The unwinding was priced once, for a single claim, at roughly two hours across three people. Nothing here multiplies the second by the first, and the reason is that the multiplication needs a figure nobody has collected — the rate at which a wrong payment is noticed at all. Unwind every one of them and the staff time dwarfs the money; unwind the few that a claimant complains about and it does not. The honest position is that the labour cost of being wrong is unmeasured, which is a different statement from small.

The grid, then, is two questions and a multiplication, and the questions are the ones an evaluation report never asks. How far can this go, and can it be taken back — asked before the accuracy number, because the accuracy number means one thing about a paragraph on a screen and another about a line on a payroll run.

What the grid does not do on its own is win the room. It supplies two of the inputs to a decision that also has to account for how variable the paths are, what tools the work genuinely requires, and what the latency budget is — and it has to be written down in a form that holds up when somebody says the workflow cannot handle anything nobody thought of.