Different models, prompts and evidence decorrelate samples; temperature does not

Reading all 38 of the errors that survive at N=7 took an afternoon and changed the roadmap. They fall into three groups that have nothing in common except being wrong. Twenty-two are cases where the clause that answers the question was never in the retrieved context at all — the retriever returned five passages, the governing clause was not among them, and every sample answered from what it had. Nine are the family the liability-cap failure belongs to, where the wording of the prompt pulls every sample towards the wrong clause. Seven require combining two clauses arithmetically, and the samples divide between two plausible readings of which one takes precedence.

None of those three groups is sampling noise, and the evidence for that is in the samples themselves. In 31 of the 38 all seven samples returned the identical wrong answer, which is the signature of a shared cause: noise produces disagreement, and seven-way agreement on a wrong value means something upstream of the sampling decided it. A shared cause is invisible to any technique whose only lever is how many times you draw from the same distribution. Temperature was settled two chapters ago and a larger N was settled in the last one. What is left is changing something that the branches do not share.

Three structural moves, and what each one breaks#

There are exactly three things you can vary across branches that constitute real information rather than noise: the model, the prompt, and the evidence. The list is short for a reason — a branch is a model, applied to a prompt, over some context, and there is nothing else in it. Each was measured on the same 200-case set against the N=5 homogeneous baseline of 79%.

The second row is the cheapest surprise on the table, and the cost movement is worth checking rather than taking on trust. Model B, from a different vendor, is priced at $1.00 per million input and $5.00 per million output against model A's $3.00 and $15.00, which is a third of the price on both axes, so a branch running on B costs about a third of a branch running on A. Moving two of five branches to it therefore removes about two-thirds of two-fifths of the bill, which is roughly a quarter, and the measured figure is a drop from $0.0310 to $0.0230 while accuracy rose six points.

What that row does not show is the maintenance: two prompts that drift apart, two rate limits to reason about, two deprecation schedules, and an aggregator that has to fold 5000000 and $5,000,000 into one key because the two models format numbers differently. The normalisation work from the aggregation chapter is what makes a heterogeneous ensemble possible at all, and a team that skipped it will find their two models never agreeing with each other on anything — which would show up as a fall-through rate rather than as a wrong answer, and would be blamed on the models.

The third row is the disappointing one, and it is worth reporting honestly rather than dropping. Three materially different framings — one that presents the clauses first and the question last, one that begins by listing the agreement's defined terms, one that asks the question and then the document — produced six more correct cases, which on 200 cases with 20 discordant pairs has a two-sided probability of 0.26. That is suggestive and not resolvable here, which is exactly the situation the previous chapter described: below about four percentage points this evaluation returns "cannot tell".

The cost went up by 22% for a reason that follows directly from the change. Prompt caching matches a shared prefix, and three different framings mean three different openings, so the span every branch has in common shrank from 2,900 tokens to about 900. What was previously billed once at the cache-write rate and read cheaply thereafter is now mostly billed at the full input rate on every branch. Varying the prompt and sharing a cache are in direct tension, and the tension is structural rather than a configuration mistake.

The fourth row is where the errors actually were. Giving each branch its own retrieval strategy — the question verbatim, the question expanded with the agreement's defined terms, a clause-type filter for liability provisions, a filter for indemnity provisions, and a whole-document keyword scan — attacks the 22 cases where the governing clause was never retrieved. Fifteen of those 22 became correct, and the mechanism is direct: at least one branch now saw the clause, so at least one branch could answer from it, where before no branch could. This is the closest analogue in this whole area to what bagging does in classical ensembling, and the analogy is offered with its limit stated: the branches differ because they were given different information, not because they were shaken, but unlike bagging the variation is in what each branch reads rather than in what it was trained on.

@dataclass(frozen=True)
class BranchSpec:
    model: str
    prompt: str
    retrieval: Callable[[str], list[Passage]]
    weight: float = 1.0

ENSEMBLE = [
    BranchSpec(MODEL_A, CLAUSE_FIRST, retrieve_verbatim),
    BranchSpec(MODEL_A, DEFINITIONS_FIRST, retrieve_with_defined_terms),
    BranchSpec(MODEL_A, QUESTION_FIRST, retrieve_by_clause_type("liability")),
    BranchSpec(MODEL_B, CLAUSE_FIRST_B, retrieve_by_clause_type("indemnity")),
    BranchSpec(MODEL_B, QUESTION_FIRST_B, retrieve_keyword_scan),
]

async def run(question: str) -> Verdict:
    async def one(spec: BranchSpec) -> Sample:
        passages = await spec.retrieval(question)
        text = await call(spec.model, spec.prompt.format(q=question, ctx=render(passages)))
        return Sample(text=text, evidence_ids=[p.id for p in passages], weight=spec.weight)

    samples = await asyncio.gather(*(one(s) for s in ENSEMBLE))
    return aggregate_weighted(samples, key_of=normalise_amount)

The evidence_ids field is the part that is easy to leave out and expensive to add later, and the reason is that it changes what a disagreement means. When all five branches read the same context, two answers differing is a fact about the model: given identical evidence, it reached two conclusions. When the branches read different context, two answers differing may be a fact about the retrieval instead: the branches were shown different documents and answered each correctly. Two branches saying $500,000 and three saying $5,000,000 is a vote in the first case and a signal that the retrieval is inconsistent in the second, and the right response to the second is to escalate rather than to count. Recording which passages each branch used lets the aggregator tell them apart, and in production about 4% of requests turn out to be the second kind.

A second vendor is not an independent observer#

The heterogeneous ensemble looks like the clean answer, and it should be sized correctly rather than assumed. Of the 31 cases where all seven homogeneous samples agreed on a wrong answer, model B agreed with them on 12. It dissented on 19, and weighted voting, which counts each branch's answer with the weight its spec carries rather than as one vote each, converted 12 of those 19 into correct answers, which is the +12 in the table.

Twelve out of 31 is a substantial share of the hardest cases to leave untouched, and the reason is not mysterious. Two frontier models, meaning the largest current-generation systems each vendor offers, trained on overlapping web-scale corpora, tuned with similar instruction-following objectives, share a great deal of what they believe about how contracts are written. The correlation between them is lower than the correlation between two samples from one of them, and it is nowhere near zero. Treating a two-vendor ensemble as though it produced independent evidence reproduces the original error one level up: the effective sample size formula from two chapters ago applies to the ensemble exactly as it applied to the repeated samples, with a smaller ρ and the same shape.

Weighting makes this worse before it makes it better. It is tempting to fit per-model weights on the held-out set — model B is better on liability questions, so give it 1.4 votes there — but 200 cases will not support fitting even two weights without overfitting, which is tuning the numbers to the accidents of these particular cases rather than to anything that holds on new ones. The previous chapter's arithmetic says why: a set that cannot resolve a four-point difference between two configurations certainly cannot resolve the smaller differences that a weight is being fitted to. And the fitted weights will not survive either vendor's next model revision, since the thing they encode is a comparison between two specific model versions. Equal weights, or weights set from a prior you can defend in a sentence — a belief about how the two models compare, held before the evaluation rather than fitted to it — are the honest choice until the evaluation set is several thousand cases.

When the budget belongs in the input instead#

The nine prompt-steering errors were fixed by changing eleven words. The extraction prompt asked for "the total liability cap", which matches the phrasing in the sub-limit clause more closely than the phrasing in the aggregate cap clause. Rewriting it to ask for "the aggregate cap on the supplier's total liability, excluding any claim-specific sub-limits" fixed seven of the nine, at N=1, for zero additional calls and 14 additional input tokens that sit inside the cached prefix.

Adding one worked example of a sub-limit contract to the prompt moved N=1 accuracy from 71% to 76%, again for no extra calls and about 180 tokens that stay cached. Those two edits together, before any ensembling, put the single-call system at 79% — the same score that five correlated samples were buying for five times the price.

That comparison is the one to keep, and it generalises into a rule about where to spend. Ensembling averages over variation in the model's response to a fixed input, so its whole domain of action is the space of responses a given input can produce. It cannot repair the input, because the input is the thing held constant across the branches. The gains available from restructuring an ensemble are real and they are second-order compared to the gains available from fixing what every branch reads.

What that licenses is a triage rule with three branches of its own. When the triage says the dominant failure is retrieval, spend on retrieval. When it says the prompt steers, spend on the prompt. When it says the model genuinely cannot do the reasoning, that is when a second model or a larger N is the right instrument, and by then it will be a small share of the remaining errors. The corollary is that the triage has to happen, and it is also the rule's residual: there is no summary statistic that produces one — every one of these decisions came from reading 38 cases, which is an afternoon, and none of them could have come from staring at the accuracy curve.

Debate designs, and what they do to the vote count#

The research direction beyond independent branches is to let the branches read each other. Multi-agent debate and mixture-of-agents designs run a first round of N independent answers, show each branch the others' answers, and run a second round in which each branch may revise. Reported gains on reasoning benchmarks are real, and the mechanism is different from voting: a branch that was wrong can be moved by seeing an argument it did not generate, which is a channel that no amount of independent sampling provides.

Two things are worth knowing before reaching for it, and both follow from the structure rather than from any measurement. The cost is at least 2N calls plus a much larger input on the second round, since every branch reads every other branch's answer, and the second round cannot start until the first has finished, so the latency is the sum of two fan-outs rather than one — the tail identity from the previous chapter applies twice and adds.

And the independence story inverts. Round-two answers are correlated with round one by design, because seeing the others' answers is the entire point, so the vote count from a debate design carries even less of the meaning that a vote count normally suggests. Confident wrong consensus is a documented failure mode here, where a persuasive wrong argument pulls the dissenters across — which is the same shared-cause failure this chapter opened with, arriving through a new channel. If you adopt it, the calibration table from the consensus chapter has to be rebuilt from scratch, because the relationship between margin and accuracy will not be the one you measured.

Everything since the cost chapter has been about voting: how to compare the samples, what their agreement means, how many to draw, and how to make them differ in ways that matter. Back on the other half of the pattern, the sectioning design has been running in the background on the contract reviewer, quietly working, with per-dimension scores that have improved every month.

It has a blind spot that no amount of measurement inside a branch can find, because the thing it misses is not in any branch. The next chapter is the report from legal that the sectioned reviewer stopped catching the findings they care about most, and why the obvious one-line fix would undo the entire reason for sectioning in the first place.