Spending a fixed call budget on the shape the task actually has

The capacity review set the number at six. Six model calls per contract review, chosen because the account's headroom divided by the peak review rate came out at 6.2 and nobody wanted to build a design that failed at the top of the hour. The question that followed was how to spend them, and the two engineers in the room had different instincts: one wanted six sections, one wanted the whole document reviewed six times with the findings voted on.

Both instincts were defensible and both had been implemented somewhere in the company, which is the situation where an argument is normally settled by whoever is more insistent. What settled this one instead is that the measurements existed by then, spread across four different evaluation runs. Collected onto one workload — the same 40-contract cross-section set and the same 300 labelled per-section findings, at a fixed six calls each — they settle the argument in about a minute of reading.

Six calls on a workload that genuinely sections#

Reading down the cross-section column shows the property that dominates everything else on this workload: recall of the finding class legal cares about is a function of how much of the document a single branch holds, and it does not care at all how many calls you spent.

The mechanism behind that column is a product of two things, and writing it out predicts most of the rows. A cross-section conflict is reported only if both of its clauses land inside one branch and that branch notices the relation between them. The second factor is roughly what one pass over the whole document achieves, which the previous chapter measured for the single-pass reviewer. So the column reads as the chance of co-location, scaled by that noticing rate. One section in six, scaled that way, lands close to the measured 8%. Two sections of 30 pages recovers 21%, since a conflict now has a one-in-two chance of falling inside one branch. The whole-document allocation recovers 49%, which is roughly what one pass over 60 pages was ever going to achieve, because there is no co-location penalty left to pay.

Six sections and three sections score the same 8%, which the co-location model does not predict on its own, and the reason is worth having because it is a lesson about reference populations. The model above treats the two clauses of a conflict as landing anywhere in the document independently. Real conflicts do not: the pair that started the previous chapter sits on page 9 and page 41, and conflicts of that kind span far more of the document than a typical section is wide. In both designs the two clauses of a typical conflict land in different branches, so widening the sections from ten pages to twenty changes nothing at all until the sections get wide enough to contain the span.

The reconciler row is the one that breaks the pattern, and it does so by spending a call on the relation rather than on the reading. Giving up one section boundary — five branches of 12 pages instead of six of 10 — cost two points of per-section score, because the branches are reading more each, and bought 63 points of cross-section recall for $0.016. Nothing else on the table has that ratio, and it is the allocation that shipped.

The third row is the one worth arguing with, because it looks like the sophisticated answer. Sampling each section twice and voting raised the per-section score from 0.91 to 0.93, and on the 300-finding set that is six findings with a two-sided probability of 0.31, which is the situation chapter 6 described: below about four points this evaluation returns "cannot tell".

It costs 73% more than the plain sectioning, and the reason is the input line rather than the call count. Three branches of 20 pages read 60,000 tokens between them, which is the whole document, and doing it twice reads 120,000. Six calls, twice the input.

Two samples is also the worst possible N for voting, and that follows from what a vote does. Two samples either agree, in which case you have learned nothing you did not already have from one, or they disagree, in which case there is no majority and the aggregator falls through to a tie-break — the outcome that chapter 4's table put at 13% of requests and that has no cheap resolution on free-text findings. A vote needs a third sample before it can do anything a single call could not.

The dotted edge is the whole point of drawing it. Once the budget is spent on samples inside sections, there is no call left for the step that recovers the cross-section findings, and the design has bought two points of a score that cannot be resolved at the price of the class it was escalated about.

A budget denominated in calls hides the decision#

Six was a call count because the constraint that produced it was a rate limit expressed in requests per minute. Costs, latency and quality do not respect that denomination, and the table makes the mismatch visible in three places.

The reconciler is one-sixth of the calls and one-sixteenth of the cost, since it reads 6,000 tokens of findings while a section branch reads 12,000 tokens of contract and the reconciler's output is short. The whole-document allocation is the same six calls and 4.7 times the money, because input tokens are what the bill is made of and six branches reading 60 pages each is 360,000 input tokens. Latency varies by a factor of two across rows that all spend six calls, because the p95 of a fan-out is the maximum of its branches and a 30-page branch has a much longer tail than a 10-page one.

Each of those three is the same failure in a different unit: a call is not a fixed quantity of anything except a call. What a call costs depends on how many tokens it carries, what it contributes to latency depends on how long its branch runs, and what it buys depends on what it was asked to do.

Budgets are worth restating in whatever unit the binding constraint actually uses before any allocation is compared. If the constraint is a tokens-per-minute quota, the budget is tokens and the whole-document row was never affordable. If it is a user-facing latency target, the budget is seconds and the two-section row is out regardless of price. If it is money, the reconciler is nearly free and the sampled-section rows are expensive. The six-call framing hid all three of those, and the only allocation it ruled out — six sections plus a seventh reconciler call — was the one closest to the answer.

A task with nothing to split#

The order-form checker answers one question: does the payment schedule in a three-page order form satisfy the net-45 term — invoices due 45 days after issue — in the governing master services agreement, the MSA that sets the terms every later order form inherits? The answer is a boolean plus the offending line item, it takes about 4,000 tokens of input, and it is verifiable, since recomputing the due dates from the extracted schedule either does or does not violate 45 days.

No allocation of five calls sections this, and it is worth walking through both ways of trying, because each one fails differently.

Splitting the input gives one branch the order form and one the MSA. Each can report its own fact, and the comparison — the question itself — remains undone until a further call reads both. That shape is a chain of dependent steps, which 11.1 placed as a different architecture with different properties: it runs serially, so three steps cost three round-trips of latency rather than one, and an error in step one propagates rather than being outvoted.

Splitting the subtask instead gives you branches asking "what is the term?" and "what is the schedule?" and "do they conflict?". The third of those needs the answers to the first two, so it is not independent of them, and launching all three at once does not make it independent — it just means the third branch answers without the information it needed. That is the same chain with the dependencies pretended away.

What the five calls do buy is repetition. Five samples, keyed on the normalised boolean and the line-item identifier, aggregated with the voter from chapter 4, and the equivalence relation is free here because the answer is already canonical once the amount is parsed.

BUDGET = 5

def allocate(task: TaskSpec) -> Plan:
    parts = independent_subtasks(task)   # returns [] when none exist
    if not parts:
        return Plan(mode="vote", n=BUDGET, key_of=task.canonical_key)

    if task.cross_part_findings_matter:
        return Plan(mode="section", parts=parts[: BUDGET - 1], reconcile=True)

    return Plan(mode="section", parts=parts[:BUDGET], reconcile=False)

The BUDGET - 1 in the middle branch is the reconciler's call, reserved before the parts are allocated rather than added afterwards, which is the ordering that the dotted edge in the diagram above was about.

independent_subtasks is the function that cannot be written generically, and writing it for a specific task is the design work this topic has been about. It returns a non-empty list only when the parts produce outputs that never need to read each other, which is a property of the task rather than of the input's length. A 220-word product description returns six parts. A 60-page agreement asked for one verdict returns none. Input size is a symptom that correlates with the answer often enough to be misleading, and the criterion is the independence.

On the order-form checker, the measured curve is 83% at N=1, 89% at N=3 and 90% at N=5, with the step from three to five worth four cases on a 300-case set and a two-sided probability of 0.42. The budget allows five and the evidence supports three, so three is what ships, and the two calls that were provisioned and not spent are headroom for the next feature rather than accuracy waiting to be collected.

Each design's blind spot, written down in advance#

The bottom row is the one that survives every change of design, and the third row is why. Both patterns fail in ways their own internal signals endorse — the sectioned reviewer's per-section score went up while cross-section recall was 8%, and the extraction voter's consensus field read 0.8 on an answer that was wrong four ways from one misreading.

That is not a coincidence of two examples. An internal signal is computed from the design's own outputs, so it can only report on the space of outputs the design produces, and both blind spots are defined by what falls outside that space. A sectioned design's per-part metric is computed over parts, and the missing finding belongs to no part. A voter's consensus is computed over samples, and the shared misreading is in every sample. Held-out labelled data is the only instrument that sees either, because it is the only one whose contents were not produced by the design.

What follows is a requirement on the evaluation set rather than on the architecture: it has to contain examples of the class the architecture is at risk of losing, and that class has to be identified from the architecture rather than from the incident report.

Five numbers cover the failures this topic has worked through, and all five come from instrumentation that costs an afternoon. Achieved concurrency, computed from time to first token rather than from task creation, distinguishes a fan-out that ran from one that queued. Absorbed retry count, read from the SDK's retry hook rather than the application log, names the cause when it queued. Cost per successful user request, split by cached and uncached input, catches the cache inversion that turned an eight-way fan-out into an 8.6× bill. The aggregation outcome distribution — unanimous, majority, plurality, tie, scattered — tells you whether the vote is doing anything, and a fall-through rate above a few percent means the equivalence relation is wrong. Accuracy against N with a paired interval, re-measured whenever the prompt or model changes, is the one that stops the pull request that raises N because the budget allows it.

The two papers worth reading next are the self-consistency result, which is where run-N-times-and-aggregate came from and states its assumptions more carefully than most summaries of it do, and the Agent Patterns Catalog entry on parallelisation, which is the clearest short treatment of the sectioning-versus-voting distinction with its execution independence conditions spelled out. Neither will tell you what your task's independent subtasks are, and that is the part to start with: take the workload in front of you, try to write independent_subtasks for it, and whichever way that goes, the allocation follows.