A rewritten instruction and a dropped tool are two other explanations

The message in the channel is short and reads like good news. Pulled the invoice-matching step out into its own sub-agent — eval went from 60% to 90% on the 50-case set. Context isolation is doing real work here, we should look at the other steps. The PR attached to it is 180 lines. The eval behind the two percentages is the step's evaluation set, a frozen collection of 50 stored inputs with known-correct answers that gets replayed on every change, so that a pass rate measured this week is comparable with one measured last week. Somewhere in the middle of those 180 lines, three things changed at once.

The subtask is the one where a bank statement line — a date, an amount, a counterparty string, a reference field — has to be matched against the merchant's open invoices, or declared unmatched. The parent loop had been doing it inline, holding the statement, the invoice list, the merchant's payment history and a running commentary on the lines processed so far, coming to about 61,000 tokens by the middle of a batch. The extracted version runs in a 3,400-token window with the one statement line and the candidate invoices.

The extraction also rewrote the instruction. In the parent loop it was two sentences buried inside a longer prompt that also covered five other steps. In the sub-agent it became a 220-word instruction with an explicit output schema and, most importantly, a tie-break rule.

-Match the statement line to one of the open invoices, or say unmatched.
-Use the reference field where it is present.
+Match the statement line to exactly one open invoice, or return unmatched.
+
+Order of evidence:
+  1. exact reference match on invoice_number  -> match, confidence high
+  2. amount equal AND counterparty similarity >= 0.8 -> candidate
+  3. amount equal only -> candidate
+
+When two or more candidates remain after step 3, prefer the invoice whose
+due_date is nearest the statement date; if two are equidistant, return
+unmatched with both ids in `ambiguous`.
+
+Return: {"invoice_id": str|null, "confidence": "high"|"low",
+         "ambiguous": [str], "reason": str}

And it dropped two tools. The child's allowlist has list_open_invoices and nothing else; the parent had also exposed search_web and write_file, which the PR description says the subtask never used.

Three candidate causes and a way to bet before you measure#

The situation is the one every backend engineer has met in a release that changed a connection pool size, a serialisation format and a retry policy together, then dropped p99 latency by 40%. The number is real. The attribution is a guess, and the guess is expensive because the next release will be planned as though it were knowledge.

Here the three candidates are the window, the instruction and the tool list, and they are not equally plausible. The evidence that settles the prior is already sitting in the failing eval runs. Of the 20 failures in the 60% arm, 13 were cases where two invoices had the same amount and the model picked the older one; the tie-break rule in the rewritten instruction addresses exactly those, and the old instruction gave no guidance at all for that case. That is a specific mechanism connecting one change to two-thirds of the observed failures, and no comparable mechanism connects the window size to them, because the duplicate-amount confusion is visible entirely within the candidate list and does not require the parent's 61k tokens to go wrong.

Reading the failures before designing the experiment is the step that turns a four-way shrug into a ranked list, and it costs an hour with the eval output open. The 20 failures in the 60% arm broke down into 13 duplicate-amount ties, 4 cases where the model matched a line to an invoice from a different merchant entirely, 2 where it returned two invoice ids in a field meant for one, and 1 timeout. The 4 cross-merchant errors are the only ones with a plausible story involving the window, since the parent's 61k tokens contain other merchants' invoice lists from earlier in the batch and the child's 3.4k do not. Four out of fifty is eight percentage points of the thirty-point gap at most, which is a ceiling on what isolation can be worth here, derived before spending anything.

The tool claim is also worth checking rather than accepting, since "never used" is a statement about traces and traces can be read. In the 50 parent runs, search_web was called 6 times, on lines whose counterparty string was an unfamiliar company name, and each of those calls consumed a turn and injected roughly 1,800 tokens of search results into the window before the model answered. Four of those six runs failed. That does not make tool removal the cause, and it does make the PR's stated reason for removing them wrong, which matters because the same reasoning will be applied to the next child's allowlist.

The experiment costs less than the argument about it#

Everything needed is already in the repository: two instruction texts, two tool configurations, a way to build either window, and a 50-case eval set with a deterministic grader. Crossing all three factors, so that every combination of window, instruction and tool list is run, gives eight arms.

ARMS = [
    Arm(context=c, instruction=i, tools=t)
    for c in ("inherited", "fresh")
    for i in ("old", "new")
    for t in ("full", "trimmed")
]

def build(arm: Arm, case: Case) -> list[Message]:
    body = INSTRUCTION[arm.instruction] + "\n\n" + render_case(case)
    if arm.context == "inherited":
        return replay_parent_upto(case) + [user(body)]
    return [system(BASE_SYSTEM), user(body)]

# 8 arms x 50 cases = 400 runs, 12 concurrent

At Ledgerline's rates this was 400 runs at about $0.031 each, so $12.40, and 38 minutes of wall-clock at twelve concurrent. The discussion in the channel about whether it was worth untangling had already run longer than that.

Averaging across the other two factors, the instruction rewrite is worth about 25 percentage points, the fresh window about 3, and the tool trim about 2. With 50 cases per arm the standard error on a single rate is around 7 points, so the 25-point effect is unambiguous and the other two are inside the noise; the honest statement is that the instruction explains the gain and the window's contribution is somewhere between nothing and small.

Note the row that never gets run in practice, and which is the whole point. Inherited context with the new instruction is 88%, one point off the shipped configuration, at a twentieth of the engineering. Nobody would have found that row without building the ladder, because the shipped design and the old design differ in three places and the space between them is not on anyone's path.

Where the untangling is not available, and what to do with the result#

Some factorial designs — the full grid of combinations laid out above — have cells that cannot be filled. If the tool trim was only possible because of the split — because the parent's tool list is assembled from earlier turns and cannot be varied independently — then fresh + trimmed and inherited + trimmed are not both constructible, and the design is genuinely a bundle. The move then is to shrink the bundle rather than abandon the question: run the two arms you can build, and separate the instruction, which is almost always independently variable because it is a string. An untangling that resolves one of three confounds — changes that moved together, so that the effect cannot be assigned to any one of them — is worth more than a shrug, since the instruction is the confound most likely to carry the effect and the cheapest to test.

It is also fine to ship the bundle. Nobody at Ledgerline was arguing for reverting a change that took the eval from 60 to 90. The objection is to the sentence context isolation is doing real work here, we should look at the other steps, because that sentence is a plan, and the plan will produce four more splits chosen by analogy to a cause that was never established. Each of those splits costs an extra model call, a boundary whose inputs somebody has to assemble and maintain, a second trace to correlate when something goes wrong, and a new failure mode where the child returns something the parent cannot use. Paying that four times over for an effect that a 220-word prompt delivers for free is the specific waste this chapter is about.

So the answer to what do you do with the sub-agent when the instruction turns out to be the whole gain has three parts and only one of them is about the sub-agent. Put the rewritten instruction wherever the subtask runs, including in the inline version, because a 25-point improvement that costs a string is the finding and it should not stay trapped inside a split. Then re-open the split as its own question with the instruction now held constant on both sides, which is a much smaller question than it was an hour earlier: does the 2-to-3-point window effect, plus whatever else the boundary gives you, justify a second call. And write the number down where the next person proposing a split will see it, because the durable output of this exercise is not the decision about this subtask, it is a locally measured estimate of what isolation is worth in this system, on this class of task, against this model.

Reverting is a real option and is under-taken. The costs of keeping a split that buys three points are mostly not the token cost, which here is negative — the child's 3.4k window is cheaper than the parent's 61k. They are the ongoing ones: a second prompt that drifts from the first when someone edits one of them, an input builder that has to be updated whenever the case schema changes, a trace that has to be joined to the parent's before anyone can debug an escalation, and a class of incident where the child returns a shape the parent does not expect. Ledgerline priced the last of these from experience at roughly one engineer-day per quarter per boundary, which is not much and is not zero, and is the sort of number that makes a three-point gain look different depending on how many boundaries you already have.

What Ledgerline did with the result was keep the rewritten instruction in both places and then re-examine the split on its own terms, which came down to a fact the eval had nothing to say about: a nightly batch has around 340 statement lines, each independent of the others, and the parent processes them one at a time. That is a case for a boundary, and it is a case about concurrency rather than about pollution. Before that argument can be made properly, though, there is a cheaper way to lose the benefit of any split, and it happens after the child has finished its work — in what it sends back.