Delete the stage the next one rewrites, then measure

The payables pipeline costs $2.85 per successfully processed invoice after the caching work from chapter 1, which at 4,100 invoices a month is $140,220 a year. Finance has asked for 40% out of it by the end of the quarter, with no loss of quality. The team's proposal, written up before anyone asked for a second opinion, is to move four of the six stages from claude-opus-5 to claude-sonnet-5 — a 40% cut in per-token price on 86% of the pipeline's tokens, reversible with a config change, deliverable in a week, and requiring a design argument with nobody.

It reaches 34%, measured, and costs 2.1 points of end-to-end correctness because the extractor is the stage least tolerant of a model change. That shortfall is worth knowing and it is not the objection. Even if the swap had landed exactly on 40% with no quality loss, it would be the wrong first move, because it reaches the number by making stages that should not exist cheaper to run. A stage nobody pays much for is a stage nobody revisits, and it still costs a boundary, a serialisation, a duplicated context and — from chapter 3 — a place for a constraint to be dropped. It also spends the mandate. You get one 40% ask a year, and spending it on the change with the smallest structural benefit means the next one has to be argued for from scratch.

Screening for stages the next one rewrites#

Before ablating anything, there is a cheap screen that runs off the handoff records and takes an afternoon. For each stage, compute how much of its output content survives into the output of the stage after it, using the same shingle comparison chapter 4 used for duplicated context. A stage whose output is largely re-derived downstream is doing work that is being paid for twice.

Two rows stand out for the same reason: validate and assemble consume a third of the budget between them and contribute content that mostly does not reach the final record. assemble is the orchestrator's compose step — the one from chapter 3 that dropped a constraint while compressing — and 88% of what it emits is a restatement of what write already produced. validate emits 4,300 tokens of prose findings, of which the pricer uses a handful of numbers.

The screen does not prove anything. It says where to point the expensive measurement, which is the ablation.

Ablation at the resolution the harness supports#

Ablating means running the pipeline with a stage removed and measuring end-to-end correctness, not the removed stage's own quality score. Removals that look catastrophic in isolation are frequently invisible end to end, because a downstream stage was already compensating. The reverse also happens.

The runs used the 400-invoice regression set, six replicates per arm, with the cluster bootstrap over invoices from chapter 5 giving the intervals. Six replicates rather than eight because each arm costs about $1,100 to run and the resolution floor was already established at roughly one point.

The − price and − extract rows are the control. They confirm the harness can see a stage that matters: removing the pricer costs 12.9 points and removing the extractor costs 42, both far outside any interval. Against that, − validate − assemble at 83.6% against 84.1% is a half-point difference on a harness with a one-point floor, for 37.5% of the cost and 87 seconds of p95.

That is close to the mandate and it is the point at which someone should be suspicious rather than pleased, because a 19%-of-budget stage that contributes half a point is either genuinely redundant or is doing something the regression set cannot see.

The defence that ablation cannot test#

The validator's author made the objection in exactly the form it always takes: the stage catches rare but expensive errors, chiefly duplicate invoices, and the regression set does not contain them.

He is right about the second part, and it is fatal to the ablation as evidence. The 400-invoice regression set was assembled from cases that once broke, weighted toward the failure modes the team was fixing at the time, and it contains zero duplicate invoices. Duplicates arrive at roughly 1 in 900 in production. To see even five of them in a sample you would need about 4,500 invoices, and to distinguish "the validator catches 80% of them" from "the validator catches 40% of them" with any confidence you would need several times that. A regression set with none of the rare cases in it will show no regression whether the claim is true or false, so running the ablation harder is not the answer — the experiment has no power against this hypothesis, meaning no realistic chance of detecting the effect even when the effect is there, at any sample size the team can afford.

The historical runs do have power, because production has been generating the rare cases all along and, since chapter 3, recording them. Ninety days of handoff records is 11,400 runs. The question to ask them is not "how often did the validator run" but "how often did the validator change an outcome", which needs a definition precise enough to compute.

def outcome_changing_flags(run_ids: list[str]) -> dict[str, dict]:
    """A flag changed an outcome if the final record differs from what the
    pipeline would have produced had the flag been absent — i.e. the flag
    either altered the priced total or blocked issuance."""
    tally: dict[str, dict] = {}
    for rid in run_ids:
        hs = load_handoffs(rid)
        flags = json.loads(read(handoff(hs, to="price").input_rendered_ref)).get("flags", [])
        priced = json.loads(read(handoff(hs, to="price").output_rendered_ref))
        final = json.loads(read(handoff(hs, to="write").output_rendered_ref))
        for f in flags:
            row = tally.setdefault(f["kind"], {"raised": 0, "changed": 0, "value": 0.0})
            row["raised"] += 1
            if f["id"] in priced.get("applied_flag_ids", []):
                row["changed"] += 1
                row["value"] += abs(priced["total"] - f["total_without_flag"])
            elif final.get("status") == "blocked" and f["id"] in final.get("block_reasons", []):
                row["changed"] += 1
                row["value"] += final["invoice_total"]
    return tally

The total_without_flag field is the one thing that had to be added to make this query possible, and it was one line in the pricer's return contract: record the total you would have produced had you ignored this flag. Without it, "did the flag change the outcome" is a counterfactual nobody can evaluate after the fact, and the analysis degrades into counting how often the validator spoke.

Over 90 days the validator raised 863 flags in three kinds. Duplicate-invoice flags: 14, of which 9 were true duplicates with a mean value of $18,400 and 5 were false positives costing an analyst about 25 minutes each. Rate-mismatch flags: 612, of which 71 changed the priced total by a mean of $340. Formatting and normalisation flags: 237, of which zero changed anything, because the write stage rewrites those fields regardless.

So 85 of 863 flags — 9.8% — changed an outcome, and those 85 were worth $165,600 plus $24,140 over the quarter, which annualises to about $760,000 against a stage costing $26,600 a year to run. The author's defence is not merely plausible; it is correct by a factor of nearly thirty, and the ablation was about to delete it.

Keep the check, delete the stage#

The value is in the checks, and the checks are not language-model work. Duplicate detection is an indexed lookup on supplier ID and invoice number, plus a fuzzy match on amount and date against 24 months of history. Rate-mismatch detection is a numeric comparison of each extracted line's unit rate against the contract rate table with a 0.5% tolerance. Both were backtested against the same 90 days of records: the deterministic duplicate check finds 11 of the 11 known duplicates in the period against the validator's 9, with 2 false positives against 5; the deterministic rate check finds all 71 outcome-changing mismatches plus 3 the validator missed. Formatting normalisation was deleted outright.

With duplicate detection moved to a pre-flight query and rate checking moved into the pricer's existing calculator tool, the pipeline runs four stages — intake, extract, price, write — at 85.9% [83.9, 87.8] correctness, $1.71 per successful document, and 198 seconds at p95. Correctness is 1.8 points above the six-stage baseline, cost is down exactly 40%, and the two checks that produced $760,000 of annual value now run in eight milliseconds with better recall than the agent that used to run them.

The general shape of this is worth carrying forward, because it recurs. A stage defended on rare-but-expensive grounds is usually defended correctly, and usually implemented in the most expensive available way. The question that separates the two is whether the check is a judgement or a comparison. A comparison against a table, an index, or a schema belongs in code, where it is exhaustive, cheap, and testable. A judgement about ambiguous text belongs in a model call, and may still not need a boundary around it.

Removing a stage without betting the quarter on an offline number#

An ablation result is evidence from 400 invoices; production is 4,100 a month with a long tail the regression set does not contain. The removals went out in three steps, and the sequencing is the part that makes a design argument survivable.

First, shadow. The four-stage pipeline ran alongside the six-stage one for two weeks on live traffic, with only the six-stage output issued. Every invoice where the two disagreed on the total or the block decision was queued for an analyst. There were 61 disagreements over 8,300 invoices; the four-stage arm was right on 44, the six-stage on 9, and 8 were cases where both were wrong in different ways. That is the number that made the change defensible to finance, because it is measured on their traffic rather than on the team's sample.

Second, canary: a release to a deliberately small slice of live traffic, watched for a while before the rest of the traffic follows. Ten percent of suppliers moved to the four-stage pipeline for a week, chosen to include the two highest-volume accounts so the sample would contain the boring cases rather than only the interesting ones. The metric watched was not correctness — nobody can score that in a week — but the AP team's exception queue depth, which is the observable proxy and which stayed flat.

Third, the rollback plan, which was the cheap part and the reason the other two were affordable. Both the deterministic checks and the validator stage stayed in the codebase behind a config flag for a full quarter, with the flag flipped and the six-stage arm re-run against the current task set once a month. The flag was removed in July after the third clean re-run.

Keeping the question answerable#

Two of the four surviving stages have the same profile the validator had — write passes through 97% of what price gives it, and intake's 900 tokens of classification could plausibly be the first turn of the extractor's own conversation. Chapter 5's chunked-loop result on the contract-review service suggests where that ends up. Whether it should is a larger comparison with its own arms, its own replicates and its own pre-registered rule, and it was not this quarter's mandate.

What makes that comparison possible next quarter rather than a rewrite argument is a small standing practice, and it is the working residue of everything in this topic. Keep cost per successful unit on the dashboard next to quality, with input, output, cache reads and cache writes separated, because the multiplier lives on the input side and a per-run figure hides the retry rate. Keep handoff records for every boundary, including the compression steps that are not agents, because attribution has to be cheap or it does not happen. Keep the simpler arm executable — not archived, executable — and re-run it against the current task set on a schedule, so that the day someone asks whether the topology earns its keep, the answer is a job you trigger rather than a project you propose. And when a new stage is proposed, ask what it will be measured on and what would have to be true for it to be removed, before it exists rather than after it has an author.

The single loop is the default because it is the design with the fewest places to lose information and the fewest tokens to pay for twice. Every boundary you add past that should be able to say what it buys, in the units the last seven chapters have been measured in, and what it costs when it is wrong.