Unchanged success and 31% cheaper is not yet a flattening win

Deleting a level is the decision most teams face, and it arrives in the opposite order from the one the previous chapter described. The tree is already there. Nobody measured anything before adding it, the engineer who did has moved on, and what remains is a shape and a plausible story about why it exists. The service here is the weekly competitive brief, and the inherited shape is three levels: an orchestrator, six segment supervisors, and eighteen research workers, one per company.

Flattening it takes an afternoon. The orchestrator writes eighteen delegation briefs directly, each carrying BRIEF_CONSTRAINTS — the shared rules about source URLs, fiscal windows and press-release pricing — and an enumerated scope of one company, and the six segment supervisors are deleted. On the 200-task regression suite the flattened tree scores 88% completion, identical to the three-level version. Tokens per successful run fall 31%, from 641k to 442k. The p95 drops 4 seconds. The pull request has two approvals and a description that reads "same quality, 31% cheaper, old topology stays behind a feature flag."

What the three numbers cover and what they leave dark#

Every number in that description is correct, and the conclusion does not follow from them, which is a distinction worth taking apart slowly rather than asserting.

Completion quality is computed over runs that produced an acceptable brief — 176 of the 200, since 88% of 200 is 176. The runs it does not cover are the other 24. The middle layer's stated purpose, according to the engineer who added it, was to catch a bad worker result before it reached the top. Catching a bad result is something that happens on a run which would otherwise have gone wrong. So the layer's work is concentrated in those 24, and in some fraction of the 176 that succeeded because something was caught and re-run. A metric that reports the population where the layer is idle, and reports it unchanged, has said nothing about the population where the layer does its job.

Setting the evidence against the claims it is being used to support makes the gap explicit rather than rhetorical.

Read the right-hand column rather than the left. Three of its four cells say the same thing in different vocabulary, and the fourth names it outright: the evidence is silent about failures. That is a stronger reading than treating the four rows as four separate caveats, because it identifies a single hole rather than a list of nice-to-haves, and a single hole can be filled by a single experiment.

The dark cells are one question asked three ways. When a worker returns a claim that is well-formed and wrong — a magnitude off by a factor, a figure from outside the fiscal window, a company profile that answers a neighbouring question — what happens to it now, and is that different from what happened to it before? In the three-level tree it landed in a segment supervisor's context alongside two sibling reports, and the contract check from chapter 3 ran at that boundary. In the flattened tree it lands in the orchestrator's context alongside seventeen siblings, and the contract check runs once, at the top, against everything at once.

Two named properties are what the question is asking about. Failure isolation is whether a bad result stays local. Trace coverage is whether, after the fact, you can say which boundary produced it. Chapter 6 made the second a shipping constraint on depth, and flattening usually improves it, since there are fewer hops and the tracing reaches all of them. Usually is not a measurement. A flattening that moves eighteen workers from the second level to the first has changed which spans exist and which parent IDs they carry, so both properties need checking in the direction they can regress, and neither appears in a completion-quality number.

The check you can build from the suite you already have#

The injection harness from chapter 5 already produces the population the regression suite is missing, so the check needs no new tasks and no new labels. It needs the same 200 tasks run with a defect deliberately introduced at a worker, in both topologies, with the same seeds, measuring what reaches the customer-visible output.

def isolation_trial(task, topology, rng) -> dict:
    """Corrupt exactly one worker return; see whether the defect escapes."""
    injected = None

    def worker_hook(boundary_id: str, payload: dict) -> dict:
        nonlocal injected
        if injected is None and rng.random() < TARGET_RATE:
            injected = Injection(boundary_id=boundary_id,
                                 claim=inject_worker_defect(payload, rng))
        return payload

    run = execute(task, topology=topology, on_worker_return=worker_hook)
    if injected is None:
        return {"applicable": False}

    return {
        "applicable": True,
        "topology": topology,
        "escaped": injected.claim.statement in flatten_text(run.final_brief),
        "contained_at": first_boundary_that_rejected(run.verdicts, injected),
        "extra_calls": run.inference_calls - BASELINE_CALLS[topology],
        "attributable": attribution_of(run.verdicts, injected.boundary_id),
    }

The injected is None guard is what keeps the trial interpretable. It fires at most once per run, so every applicable run carries exactly one corruption and the escape question has one subject. A run with two injected defects would give an ambiguous answer — contained one, escaped the other — and there would be no principled way to score it.

Three fields carry the result. escaped is the customer's question and the one that decides the flattening. contained_at names the boundary that rejected the corrupted return, which is what turns "the tree recovered" into "this hop recovered", and that distinction is the whole finding of this chapter: once you know which hop caught something, you can ask whether that hop needed to be an agent. attributable is the trace-coverage check — given the verdict stream from a failed run, can a query name the boundary the defect entered at?

That third field is the same attribution coverage chapter 6 defined as a rate over failed runs, computed here over the injected population instead of over incidents. The substitution is what makes it affordable. Waiting for enough real incidents to estimate attribution coverage takes a quarter, because failures are rare by design; injecting defects produces as many as the trial rate allows, in an afternoon, and the property being measured is the same one.

Running that over the 200 tasks in both topologies produced the comparison the pull request needed.

The escape row is the one that would have shipped a regression. Flattening tripled the fraction of corrupted claims reaching the final brief, from 22% to 61%, while every number in the pull request description stayed true. The mechanism is not subtle, and it is the same one that produced chapter 5's recovery gap. In the three-level tree a corrupted return faced a contract check with three companies in scope, where a stale as_of or an out-of-scope name is a set comparison against a list of three. In the flattened tree the same check runs once over eighteen companies as part of a step that is also writing the brief, and the corrupted claim is one of about two hundred.

The third column is where the chapter's finding lives, and it is a dominance rather than a trade. Set it against the first column row by row: identical completion, fewer tokens, the same p95, better attribution coverage, and an escape rate of 24% against 22% — a difference small enough that no experiment of this size would resolve it. A column that matches or beats another on every row settles a question rather than posing one. Containment in the original tree came from judge_boundary running at the segment boundary, and not from the segment supervisor's judgement; the supervisor was the thing that happened to be standing where the check ran. Running the same deterministic check on each worker return restores the escape rate at a cost of 13k tokens per run and no inference at all, since the check is a schema validation and two set comparisons.

The fourth row records where containment happened, and it moves in a direction worth naming. In the three-level tree the median containment was at the segment supervisor; in the flattened tree it was at the orchestrator, which is the top; in the third column it is the worker return check, which is lower than either. Containment migrating downward is the improvement, and the reason is the same set-comparison argument one step further on. A check at the segment boundary compares a return against a scope of three companies. A check on a worker return in the flattened design compares it against an enumerated scope of one, so an out-of-scope name has nowhere to hide and a stale as_of is a single date against a single window. The earliest possible check is also the tightest one, and it is the cheapest, since a defect caught at the worker costs one worker re-invocation rather than the discard of a summary built on top of it.

The p95 row confirms that none of this costs time: 139s in both flattened columns. A deterministic check adds no inference call and no model wait, so adding it to eighteen worker returns is invisible in wall-clock. That is the practical reason the third column is available at all — a containment mechanism that had cost seconds would have had to be traded against the latency the flattening bought.

The 13k is worth reading correctly, because a deterministic check that costs tokens sounds like a contradiction. The checker itself makes no model call. What the tokens buy is the consequence of the check: a rejected worker return gets re-invoked, and that re-invocation is worker tokens. So the figure is retry cost caused by catching more, and an arm that caught nothing would show none of it.

The flattening is therefore a win once that check moves, and it was a regression without it. Nothing in the original evidence distinguished those two cases, and that is the practical lesson: the pull request was neither right nor wrong, it was undetermined, and the three numbers in its description could not have told anyone which.

Testing the claim that the layer caught things#

The middle layer's author made a specific empirical claim, and specific empirical claims can be checked against traces that already exist rather than debated. Six weeks of production traces cover 240 runs. Each run invokes six segment boundaries, one per segment supervisor, so those weeks contain 240 × 6 = 1,440 segment-boundary invocations. The question is how many of those rejected a worker return, what happened next, and whether the rejection would have been made anyway somewhere cheaper.

def containment_events(verdicts: list[BoundaryVerdict]) -> dict:
    by_hop = defaultdict(list)
    for v in verdicts:
        by_hop[(v.run_id, v.boundary_id)].append(v)

    rejected = {k: vs for k, vs in by_hop.items()
                if any(not v.brief_satisfied for v in vs)}
    repaired = {k: vs for k, vs in rejected.items()
                if any(v.brief_satisfied for v in vs)}
    return {
        "invocations": len(by_hop),
        "rejected_once": len(rejected),
        "repaired_on_retry": len(repaired),
        "abandoned": len(rejected) - len(repaired),
        "reasons": Counter(f.split(":")[0]
                           for vs in rejected.values()
                           for v in vs for f in v.constraint_failures),
    }

The grouping key is the pair of run id and boundary id, which is chapter 3's distinction between attempts and boundaries expressed in three lines. A hop that failed once and passed on retry appears in rejected because any of its attempts failed, and in repaired because any of them succeeded, so it is counted as a containment event and not as a loss. abandoned is the difference, and it is the only outcome that cost a run.

Over those 1,440 invocations, 96 rejected at least one return and 71 recovered on the retry. The reason histogram splits them: 62 were schema failures, and 34 were scope or constraint failures, with a handful counted in both.

That split decides the argument, and it decides it by subtraction. The top-level contract check in the flattened design performs schema validation on everything it receives, so the 62 schema failures were never dependent on the middle layer at all — they would have been caught one level up. What remains as the layer's actual contribution is the 34 scope and constraint failures. Replaying those 34 returns through the flattened topology showed 21 of them reaching the final brief unchallenged. Twenty-one escapes over 240 production runs is 8.75%, which is a real rate of runs carrying a defect the deleted layer used to stop.

That 8.75% is a floor rather than a total, for three separate reasons, and saying so is not a hedge. The 62 schema failures were excluded from the replay by an argument rather than by measurement. The population is production runs that happened to contain a rejection at all, so any defect that no boundary rejected in either topology is invisible to this count. And a replay is not a re-run: it puts a recorded return through a different topology, which tests the checks and not the rest of the run. A number built that way can only understate.

A feature flag needs a signal behind it#

Keeping the old topology behind a flag reads as prudence, and on a latency regression or an error-rate spike it would be. It does not work here, and the reason is worth following link by link, because the flag is the part of the pull request that felt most responsible.

A flag is a mechanism for acting on a signal. The regression this flattening risks is a rise in the escape rate. The escape rate is not on a dashboard, and it cannot be, because production has no injected defects — so there is nothing to count escapes against. A corrupted claim that reaches a brief becomes a sentence in a document a customer reads. It comes back, if it comes back, as a support ticket weeks later, phrased as a complaint about accuracy rather than as an incident with a run id. Nothing in the suite, the dashboards or the alerts will ever tell anyone to flip that flag, so the flag will sit at its new value until someone deletes the dead branch during a cleanup.

What replaces the flag is a signal and a written trigger. The signal is a canary — a small slice of live traffic deliberately run under harsher conditions than the rest, so that a problem shows up there before it shows up everywhere. Here that is 2% of production runs executing with an injected defect, with the escape measured automatically, and the escape rate promoted to a service level indicator — an SLI, the headline number a team publishes for how well a service is doing its job — with the same standing as latency. Attribution coverage is the second SLI, computed weekly over failed runs; it is the property chapter 6 capped depth on, and flattening improved it from 71% to 94%, which is a gain worth defending rather than assuming.

The trigger is written before the merge, names a threshold and an owner, and says what happens when it fires: if the escape rate exceeds 30% over any rolling 200 runs, restore the per-worker contract check and re-measure before considering topology. Note what that sentence does not say. It does not say restore the middle layer, because the middle layer was never the thing doing the work. A rollback condition written afterwards is a preference, and it will lose to whoever is arguing at the time.

The operating loop this topic has been assembling is now complete enough to run against a system you did not build. Count the boundaries and name what each one carries. Define what clean means at each hop and instrument all three definitions, keeping attempts and boundaries in separate denominators. Compute the naive compounding figure, then look for shared-origin failures the multiplication cannot see. Cap depth at what your tracing can attribute, and write the conditions that would raise the cap. Require a measured defect rate before adding a level, and require an escape-rate measurement before removing one.

Running that loop on this tree produced an answer that neither side of the review had proposed: delete the level, keep the check, and pay 13k tokens for what two supervisor calls had been charging for. The trees worth shipping are the ones where every level has an answer to what it is for, and the answer is a number someone took before the level went in.