A sectioned review cannot see what crosses its boundaries

The contract reviewer had been re-cut. Agreements that used to run 30 pages were running 60, the cost accounting had shown input tokens dominating the bill, and splitting the document into six page-range sections with one branch each brought total input back to roughly what one pass used to cost. Each branch returned structured outputs — a JSON array of findings with clause identifiers, categories and amounts — and a merge step concatenated them into the report. Per-section quality, scored against the labelled finding set, went from 0.86 to 0.91. Every number the team watched improved.

One class of finding stopped appearing, and a single pair of clauses shows what it looks like. Clause 6.4, on page 9, gives the customer the right to terminate for convenience on 30 days' notice. Clause 19.2, on page 41, commits them to a minimum spend of $1.2M across 36 months payable whether or not the agreement is terminated. Either clause is unremarkable on its own and the pair is a trap.

No branch had both clauses in its context#

The branch that read pages 1 to 12 saw clause 6.4 and correctly recorded a termination-for-convenience right with 30 days' notice. The branch that read pages 37 to 48 saw clause 19.2 and correctly recorded a minimum-spend commitment of $1.2M. Both branches did their job.

The finding legal wanted is a relation between two facts, and a relation has two terms. Producing it requires holding both terms at once, because the judgement is about how they interact and neither one on its own carries any information about the other. No single branch ever held both. So the finding is not in the space of outputs the design can produce, whatever any branch does with the material it has.

This is worth separating from the failures the earlier chapters dealt with, because it responds to none of the same remedies, and the reason is that those remedies all operate inside a branch. A branch that misreads its own section is a quality problem inside a branch, and sampling that branch five times and voting is a reasonable response, since the correct reading is available to it. A branch that queues behind a quota is a throughput problem, and a semaphore fixes it. Here every branch is correct, fast and unanimous, and the design still cannot produce the finding.

Rewriting the branch prompt to say "also flag any conflicts with other parts of the agreement" produces branches that flag nothing, or worse, branches that speculate about clauses they have not read. That is the belief worth naming, because it is the first thing anyone tries: an instruction reads like a capability, and it is not. An instruction can direct attention within the context a branch has; it cannot add context. The chapter on decorrelation established that ensembling cannot repair the input; this is the same principle applied to a partition — averaging over branches cannot recover information that the partition deleted.

Chapter 1 named the two flavours of sectioning, and the distinction turns out to carry the entire failure. When the subtask splits and every branch reads the whole input, cross-cutting relations are visible to any branch that thinks to look for them, and the earlier dimension-sectioned reviewer did catch some of them for exactly that reason. When the input splits, the branches are blind to each other by construction, and that blindness is the same property that made the design cheap. The two are one decision, not two, which is why the fix cannot simply be to undo the blindness.

A reconciliation pass over the structured outputs#

The fix does not require any branch to read more. It requires one more call that reads what the branches produced rather than what they were given, and the reason it is cheap is a size argument: six sets of findings are two orders of magnitude smaller than the document they came from, so a call over all six findings arrays is a small call even though a call over the document would not be.

{
  "section": 2,
  "pages": [13, 24],
  "findings": [
    {
      "clause": "6.4",
      "page": 9,
      "category": "termination",
      "summary": "Customer may terminate for convenience on 30 days' written notice.",
      "party": "customer",
      "amount": null,
      "duration_days": 30,
      "conditions": [],
      "references": ["3.1"],
      "defined_terms_used": ["Agreement", "Notice Period"]
    }
  ],
  "obligations_created": ["customer:notice:30d"],
  "obligations_referenced_but_not_defined": ["Committed Spend"]
}

Two fields in that schema exist only to serve the reconciler, and neither is something a branch needs for its own report. references records every clause number a branch's section points at, so the reconciler can follow a pointer into a section that a different branch read — a branch cannot follow that pointer itself, because the target is outside its pages, but it can record that the pointer exists. obligations_referenced_but_not_defined records the terms a branch encountered without a local definition, which is the single strongest signal that a clause depends on something outside its own section. The branch covering pages 13 to 24 emitted Committed Spend on this contract, and that entry alone would have surfaced the conflict.

The reconciler is given the six findings arrays and a rubric of conflict types — a right granted in one place and removed in another, two clauses stating different values for the same quantity, an obligation whose trigger is defined in a section that never mentions the obligation, a defined term used before it is defined with a different sense. Each of those four types is checkable against a field the schema already carries, which is what keeps the reconciler grounded rather than speculative. A right granted and removed is a pair of findings whose category and party line up against each other. Two values for the same quantity is a comparison of amount fields across findings that name the same subject. An obligation whose trigger sits elsewhere is exactly what obligations_referenced_but_not_defined records on one side and obligations_created records on the other. And a defined term used in two senses is a collision in defined_terms_used. The schema was designed backwards from the rubric, which is why the reconciler can work without the document.

It is instructed to return only pairs, each with both clause identifiers and a one-sentence statement of the incompatibility, and to return an empty array rather than reaching for something marginal. On this contract it returns the 6.4 and 19.2 pair with the sentence "termination for convenience does not discharge the committed minimum spend, so exercising 6.4 leaves the customer liable for the balance of 19.2".

At 4,800 tokens of findings plus a 1,200-token rubric in and 700 tokens out, the call costs $0.0285, against $0.243 for the six branches that produced its input. It is serial after the fan-out, so it adds about 4.1 seconds to a 31-second wall clock. The rest of the design does not change.

What the two obvious alternatives cost#

The proposal that arrives first in every one of these conversations is to give each branch the full contract while asking it to report only on its own section, on the grounds that it restores the missing context with a one-line change.

It does restore the context, and it pays for it twice. It converts a design whose input cost was flat into one where input scales with the number of branches, which is the shape chapter 3 measured on the dimension-sectioned reviewer before the cache stagger. And it puts every branch back to reading 60 pages to write findings about 10, which is the long-context reading that sectioning was adopted to escape, so the per-section quality gain goes with it.

Read the first row against the last column before anything else, because it explains the rest of that column. Six branches each reading the whole 60,000-token document is 360,000 input tokens, where the sectioned design's six slices add back up to the document. The reconciler design adds only the findings and its rubric on top of the same 60,000, which is why its input line barely moves.

The full-document column is the expensive way to end up in the middle. Cross-section recall — the share of the 40 known conflicts that a design actually reports — rises from 8% to 33% rather than to anything like the single pass, since a branch that can see clause 19.2 has still been told to report on pages 13 to 24 and mostly does what it was told. Cost rises 4.7 times. Per-section score falls back most of the way to the single-pass number. Every axis moves in the direction the sectioning was supposed to move away from, in exchange for a third of the finding class it was supposed to recover.

The reconciler's 73% is above the single pass, which surprised the team and has a mundane explanation. The reconciler compares six dense structured summaries rather than searching 60 pages of prose, so the comparison it performs is much easier than the one the single-pass reviewer was doing implicitly. A single pass has to notice a relation while reading; the reconciler is handed both terms already extracted and asked only whether they conflict.

The evaluation scored 0.91 the entire time#

The per-section evaluation is 300 labelled findings, each belonging to a section, scored branch by branch. A cross-section conflict has no section to belong to, so no annotator ever produced a label for one, so the metric had no way to move when the class disappeared. It went up, correctly, because the per-section work genuinely improved. An evaluation that cannot express a failure will report success through it.

The change is to split the evaluation to match the failure modes the design has rather than the structure the design happens to use. Legal's folder of historic redlines yielded 40 contracts with known cross-section conflicts, which became a second held-out set scored on recall and precision for that class alone, reported next to the per-section number and never averaged into it.

Averaging would have buried it, and the arithmetic of why is short enough to check. The per-section set carries 300 findings, more than seven times as many items as the cross-section set's 40 cases. Between the single pass and the sectioned design, the per-section score gained five points across the large set while cross-section recall lost forty-seven points across the small one. Blend the two and the larger slice's gain nearly cancels the smaller slice's collapse, so a combined score moves by under two points however badly the class fails — and it is the class legal values most.

The generalisation is that a partition of the work implies a partition of the errors, and the errors that fall between the parts need their own measurement. The chapter on the accuracy curve made a related point about what a 200-case set can resolve; this is the prior question of whether the set contains the failure at all. Whenever you change the shape of a pipeline, the honest move is to ask which class of output the new shape can no longer produce, and to go looking for labelled examples of it before the users find them.

Where the reconciler's own recall stops#

The reconciler's precision, the share of what it flags that is a real conflict, needs stating alongside its recall, because a step that reads summaries rather than documents is well placed to invent relations. It flags an average of 1.8 conflicts per contract of which 0.6 are spurious — a pointer followed into a clause that qualifies rather than contradicts, most often. Requiring both clause identifiers to exist in the findings, and rejecting any pair where the reconciler cannot quote both summaries, removed about half the spurious flags without changing recall. That rule works because it converts a judgement into a lookup: a pair whose terms are not both present in the input is not a conflict the reconciler could have grounded, whatever it wrote about them.

The remaining 27% of the class is a real limit rather than a tuning problem, and it follows from where the reconciler sits. It can only relate facts that some branch wrote down, so its recall is capped by per-branch extraction recall. A conflict where one of the two clauses was never emitted as a finding is as invisible to it as it was to the branches, since the reconciler never sees the document and cannot notice an omission in a list it has no way to check. Enriching the schema helps at the margin, because more fields mean more chances that a relevant clause leaves a trace, and it cannot close the gap; what closes it is per-branch recall, which is a different piece of work.

Both halves of this topic now have their blind spot written down. Sectioning loses what crosses its boundaries and needs a reconciliation step to recover part of it. Voting spends N calls on one question and buys no coverage at all, and its returns flatten where the correlation says they will.

What has not been done is put them on the same workload with the same money and compare the result. The next chapter takes a fixed budget of six calls per request, spends it every way that is available — six sections, five samples and a judge, three sections sampled twice, three samples plus an adaptive top-up — and reports quality, latency and true cost for each, so that the choice between them is made from a table rather than from a preference.