Parallel fan-out buys wall-clock and pays in duplicated input

Ledgerline's reconciliation batch starts at 02:10 and has to be finished before the merchant dashboards recompute at 03:00. A typical night is 340 bank statement lines, each of which needs matching against that merchant's open invoices. Run inline, one line after another, each call takes about 3.1 seconds and the batch takes 1,054 seconds, which is 17.6 minutes and comfortably inside the window. The night it stopped being comfortable, a payout provider had backfilled six weeks of delayed settlements and the batch was 4,100 lines, which is 3 hours 32 minutes, and the dashboards recomputed against a half-reconciled ledger.

The fix is the obvious one and it works. Twelve branches running concurrently, each handling one statement line in its own window, brought the 340-line batch to 104 seconds of branch time plus 11 seconds of joining, and the 4,100-line night to about 21 minutes. That is a real thing the split bought, and it is a completely different thing from what the previous four chapters have been about. The invoice matcher's window was already clean; the extraction in chapter 4 established that, and established that the cleanliness was worth about three percentage points. Nothing about running 340 clean windows at once makes any of them cleaner. What it changes is when they finish.

The ledger for concurrency has different line items#

The unit of comparison is a scatter-gather across a service fleet, which most backend engineers have built at least once: you split a request into N sub-requests, you pay for the fan-out, and you get back the latency of the slowest branch rather than the sum, once a join — the step that waits for every branch and reconciles what they returned into one result — has collected them. The line items here are the same ones, with one addition that has no counterpart in a normal service fan-out.

Each branch re-sends the shared inputs. The invoice matcher's 3,400-token window is about 2,600 tokens of material that is identical across every line in the batch — the system prompt, the 220-word instruction from chapter 4, and the merchant's open invoice list — and about 800 tokens that are specific to the line. Inline, the shared 2,600 sits in the window once and every subsequent line is 800 more tokens on top. Fanned out, the 2,600 is sent 340 times, which is 884,000 tokens of duplication per batch, or $2.65 at $3 per million input tokens.

Prompt caching — the provider storing the processed form of an identical leading stretch of a prompt, so that later calls carrying the same stretch are billed at a reduced rate for it — changes that number by an order of magnitude and is the reason fan-out is affordable at all. If the shared 2,600 tokens are arranged as a stable prefix, the first branch writes the cache and the other 339 read it, billed at roughly a tenth of the input rate, so the duplication cost falls from $2.65 to about $0.27 per batch. The arrangement is a constraint on how you build the child's input rather than a setting you turn on: anything that varies per branch has to come after everything that does not, so a builder that interpolates the statement line into the middle of the instruction destroys the shared prefix and quietly restores the full $2.65.

PREFIX = SYSTEM + "\n\n" + INSTRUCTION + "\n\n" + render_invoices(open_invoices)
#        ^ identical for every branch in the batch; cache_control marks it

async def branch(line: StatementLine) -> Proposal:
    return await model.complete(
        system=[{"text": PREFIX, "cache_control": {"type": "ephemeral"}}],
        messages=[user(render_line(line))],     # ~800 tokens, varies per branch
        tools=[list_open_invoices],
        max_tokens=300,
    )

sem = asyncio.Semaphore(12)                     # provider rate limit, not a tuning knob
proposals = await gather_with(sem, [branch(l) for l in lines])
assignments, contested, failed = join(proposals)

The concurrency limit in that snippet is set by the provider's request rate rather than by anything about the task, and it is the parameter that most often turns a promised speedup into a disappointing one. Ledgerline's tier allows 60 requests per minute on this model, so twelve concurrent branches at 3.1 seconds each is already saturating it; the 340-concurrent column in the table below is arithmetic rather than something they can run, and asking for it produces 429s, the HTTP status a provider returns when it is refusing a request for exceeding the rate limit, which the retry policy then converts into latency. A fan-out design that has not checked the account's rate limit has not established its wall-clock claim, because the limit, and not the branch count, sets the floor.

The failure row is the one that surprises people, and it is arithmetic rather than a property of models. A per-call failure rate of 0.4% — rate limits, transient 500s, an occasional call that overruns its timeout — is invisible when you make one call and is close to certain across 340. The probability that all 340 branches succeed is 0.996 raised to 340, which is 0.26, so three nights in four have at least one branch that did not come back. Sequential execution has the same per-call rate and the same expected number of failures, but it meets them one at a time inside a loop that can retry in place with the failure fresh in its window; the fan-out meets all of them at the join, after the fact, with nothing but a branch id.

What the join has to do that the sequence did not#

Every branch produces its answer without seeing any sibling's answer, which is the point and is also the source of the one line item that a service fan-out does not have. Two statement lines on the same night, both for £1,240.00 from the same counterparty, were each matched to invoice INV-8841 by their respective branches. Neither branch was wrong on its own evidence. Run in sequence, the second call would have seen that INV-8841 was already consumed and would have looked further, or returned unmatched with a reason. Run in parallel, the ledger gets a double-assignment that a downstream job later turns into a credit note nobody can explain.

Six such collisions turned up per 340-line batch once someone looked for them, which is 1.8% of lines and was not visible in the eval set from chapter 4 because that set graded one line at a time. The repair is the one that scatter-gather designs usually converge on, which is to demote the branches from deciders to proposers. Each branch returns its ranked candidates with confidences rather than a single assignment, and the join performs the assignment across the whole batch, which is a small matching problem with a deterministic solution and no model involved. That change cost eleven lines in the join and removed the collision class entirely, and it is worth noticing that it works by putting the cross-branch dependency somewhere it can be handled exactly, rather than by giving any branch more context.

The other thing the join owns is the partial answer, and the policy is a product decision that people try to make as an infrastructure decision. Retrying the whole batch doubles cost and latency for one bad branch. Retrying only the failed branches adds one branch's latency plus backoff, which is about 8 seconds, and is right when the failure was transient. Degrading — marking the line unreconciled and routing it to the human queue — is right when the failure is the model refusing or exceeding its budget, since retrying that produces the same result more slowly. Ledgerline retries twice with jitter, a randomised delay added so that a batch of simultaneous retries does not arrive simultaneously a second time, and then degrades, which sends about 17 lines a week to a human, and the number is small because the batch is large; the same policy on a five-branch fan-out where one branch is the important one is a different proposition entirely.

The test is data-flow, and it is not about teams#

The question of whether a subtask belongs in a branch gets answered in most design reviews by describing the subtask's responsibility, which is an organisational answer to a data-flow question. Two checks settle it properly. First, can this subtask's input be written down without reading any sibling's output — not "does it usually not need it", but is there a builder, of the kind chapter 2 described, whose parameters can be filled before any sibling has run. Second, would you accept the result if the siblings had run in the opposite order, or in a random order, or all at once. The second check catches the invoice collision, because the answer is no: line 12 matching INV-8841 and line 207 matching it are not interchangeable outcomes, and the sequential version was silently relying on order to break the tie.

Both checks are about the data. A subtask can be owned by a different team, have its own evaluation set, its own prompt file and its own on-call rotation, and still fail both, in which case it is a well-organised sequential step. A subtask can be a hundred lines of the same prompt with a different index and pass both, in which case it is a branch. The organisational story and the data-flow story are independent, and only one of them determines whether concurrency is available.

There is a floor on N below which none of this pays. Three branches of 3.1 seconds each save at most 6.2 seconds against sequential, and they cost a join, a partial-failure policy, three input builders' worth of maintenance, and a trace that now has four spans to correlate instead of one. Ledgerline's rule of thumb, arrived at by having built both, is that fan-out is worth its overhead somewhere above a dozen branches or above a minute of sequential time, and below that the honest answer is to leave the loop alone. The 4,100-line night is unambiguously above the line; a three-way fan-out over the sections of a short document is not, and it is the shape that arrives most often in design documents.

Concurrency and isolation are also independent of each other, which is worth stating plainly because the two get bundled under "we used sub-agents". A branch that runs concurrently while receiving the parent's whole transcript is parallel and not isolated; it will be fast and it will carry every problem chapter 1 described. A reviewer with a hand-built 4k window that runs strictly after the plan it reviews is isolated and not parallel; it buys nothing in wall-clock and everything in what it is conditioned on. A design that claims both owes evidence for both, and the evidence is different in each case: an inherited-fraction measurement for the first, and a dependency argument plus a latency measurement for the second.

That gives the positive standard a split proposal has to meet, and it is short. Show a window that is measurably smaller than the single loop's would have been at the same point, with the inherited fraction to back it. Or show that the branches are independent by the two checks above, and a wall-clock win large enough to pay for the join, the duplicated input and the partial-failure policy. Or show a quality difference on the subtask, measured the way chapter 3 measured it, with the instruction and inputs held constant. Any one of the three is enough. What is not enough is a diagram in which each box has a name, and the next chapter is about a proposal that offers exactly that and is otherwise carefully argued.