Sampling diversity and error correlation

The assumption underneath majority voting is that when one sample is wrong, another sample is not wrong in the same way. Everything the technique buys comes from that, and everything it fails to buy comes from its absence. So the question that decides whether a fan-out will pay is not what N is or how the votes are aggregated — it is where the difference between the samples comes from, and whether the difference reaches the part of the answer that can be wrong.

There are two ways to get this wrong and they look identical in a dashboard. The first is producing no diversity at all, which happens more often than it should because the sampling configuration usually lives somewhere other than the code doing the voting. The second is producing plenty of diversity in wording, reasoning and intermediate steps while every sample converges on the same incorrect final value. In both cases the vote runs, the logs look healthy, and the accuracy does not move.

Separating those two failures needs one distinction held firmly, because the words for them are used interchangeably everywhere else. Diversity is a property of the answers: how many different strings came back. Decorrelation is a property of the errors: given that two samples are wrong, whether they are wrong in the same place and the same way. Diversity is necessary for decorrelation, since identical answers cannot fail differently. It is not sufficient, because two answers can differ in every word and still carry the same wrong number. Almost everything that goes wrong with a fan-out comes from measuring the first and assuming the second.

Greedy decoding returns one sample N times#

Greedy decoding takes the argmax at each step — the single highest-scoring next token, with nothing left to chance — so with the same weights, the same prompt and the same server it should return byte-identical output every time. In practice it usually does not, quite. Floating-point addition is not associative, batched inference sums partial products in an order that depends on what else is in the batch, and a mixture-of-experts model, one whose layers send each token to a small subset of specialist sub-networks rather than through all of them, may route a token differently depending on batch composition. Running the same prompt 2,000 times at temperature 0 against a hosted endpoint, one extraction team saw three responses that differed from the other 1,997, all three in a single numeric field on long invoices where two candidate values were nearly tied.

That is worth knowing about so you do not conclude your logging is broken, and it is worth being clear that it is useless as a source of diversity. The variation appears at a rate of 0.15% and appears exactly where the model was already indifferent, which is not where the errors are. Treating it as sampling is like hoping cosmic rays will explore your search space.

Both halves of that dismissal come straight out of the run. Three responses differing out of 2,000 is three in two thousand, which is the 0.15%, and at that rate a fan-out of five samples would almost always return five identical strings. The second half matters more. The field that moved was one where two candidate values were nearly tied, and a near-tie is a position where the model has no strong opinion — the two scores sit close together, so which one wins gets decided by an accumulation order that has nothing to do with the invoice. Errors worth voting away are the opposite shape: the model holds a confident opinion and the opinion is wrong. A confident opinion means a wide gap between the top score and the next one, and a wide gap is precisely what a reordered floating-point sum cannot close. The variation is real, and it is concentrated in the positions where the answer barely matters.

The same absence of usable variation happens more quietly at settings that are not zero. A top_p of 0.1 left over from a formatting-stability fix does the same job without announcing itself, admitting just enough variation to make the samples differ in whitespace while the answer token stays pinned. So does a heavily constrained decoder, a JSON schema tight enough that only one field ordering is legal, or a prompt whose few-shot examples — the handful of worked cases pasted in above the real input — make one answer overwhelmingly likely.

What temperature buys and what it charges for it#

Temperature rescales the logits before the softmax, flattening the distribution as it rises, so higher values put more probability on tokens the model ranked below first. The effect on a batch of samples is easy to state and easy to get wrong in practice, because two quantities move in opposite directions. Individual samples get worse, since you are increasingly likely to draw a token the model ranked second or fifth. The set of samples gets more varied, so the chance that at least one of them is right goes up. Voting sits between those two and peaks somewhere in the middle.

The two named pieces of that first sentence carry the whole trade, so they are worth unpacking. A logit is the raw score the model assigns to each candidate next token before anything is normalised, and the softmax is the step that turns a list of such scores into probabilities, by exponentiating each one and dividing by the total. Temperature divides every logit by the same value before that exponentiation. Dividing by a number larger than one shrinks the gaps between the scores, and shrunken gaps come out of the exponential closer together, so the probabilities even out and tokens the model ranked below first start getting drawn. Dividing by a number smaller than one stretches the gaps the other way, which is why lowering temperature sharpens the distribution, and why the limit at zero is the argmax: stretch the gaps far enough and the top-ranked token takes essentially all of the mass.

Swept across five settings on an extraction task with 900 cases and five samples each — coverage meaning the fraction of cases where at least one of the five was correct, which is the ceiling any aggregation rule could reach:

The top row is the one to read first, because it is a consistency check on everything below it. At temperature 0.0 the samples produced 1.00 distinct answers, meaning all five came back as the same string, and once that is true the last three columns are forced to agree. If five samples are one answer repeated, then "this sample is correct", "at least one of the five is correct" and "the mode is correct" are three descriptions of a single event, so per-sample accuracy, coverage and majority-vote accuracy have to read the same figure. They all read 84.0%. A fan-out at temperature zero is five calls billed for one call's worth of information, and the row says so without any commentary.

Coverage rises steeply through 0.7 and then flattens, while per-sample accuracy erodes slowly and then falls off. The vote's peak at 0.7 is where those two curves cross in the useful direction, and at 1.3 the vote has given back everything it gained: the samples are diverse, the correct answer is present in 93.1% of cases, and it is no longer the plurality often enough to win. The gap between coverage and vote accuracy — 92.6% against 88.7% at the best setting — is four points of headroom that a better selection rule could in principle recover.

The turnover at the top of the sweep has a mechanism rather than a shrug, and the mechanism is the plurality condition. A vote returns whichever answer has the most samples behind it, so it wins when the correct answer holds a bigger pile than any single wrong answer does. Raising temperature does two things to those piles at once. It makes the correct answer reachable on cases where it was previously unreachable, which is the coverage column climbing. And it thins the correct pile, by scattering draws that would otherwise have joined it. Through the middle of the sweep the first effect dominates. Past it the correct answer is already reachable nearly everywhere it is ever going to be — coverage peaks at 1.0 and then falls back — while per-sample accuracy is dropping several points a step, so the only thing further spread can do is take draws off the winning pile. The vote column tells that story on its own: it begins at 84.0% and returns to 84.0%, and the entire excursion in between is the range over which the two effects were not yet cancelling.

The four points is 92.6% minus 88.7%, and naming what the difference is made of matters more than the subtraction. Coverage counts cases where the right answer appeared somewhere among the five; vote accuracy counts cases where it also won the count. The difference is therefore exactly the set of cases where the model generated the right answer and the aggregation rule discarded it. That makes it the largest gain any better selector could deliver without touching the generator at all — and an upper bound rather than a forecast, since no real selector picks out the right answer every time it is present.

The important column for present purposes is the middle one. At temperature 0.3 the five samples produced 1.31 distinct answers on average, meaning that on most cases they produced one answer five times. Whatever the vote was doing on those cases, it was not aggregating independent evidence. A quick way to catch this before you have accuracy numbers is to log the distinct-answer count and look at the distribution rather than the mean, since a bimodal picture — mostly 1, occasionally 4 — tells you the sampling is only exploring on the cases the model already finds ambiguous.

Two steps are compressed into that reading and both are worth having. The first goes from the mean to the shape, and it works because of the floor. A distinct-answer count cannot fall below one, so every case that produced two or more distinct answers has to be paid for by cases sitting at exactly one. A mean of 1.31 leaves under a third of a count per case to spend above that floor, which is nowhere near enough to fund many cases at three or four, so most of the set is one answer repeated five times. The second step is why the mean conceals this rather than reporting it: the same average is produced by every case being mildly varied and by a small minority being wildly varied, and those two situations call for opposite responses. A histogram separates them and an average cannot.

Diversity is not the same as decorrelated errors#

Distinct answers are cheap to count and only half the question. Two samples can differ in their reasoning, their wording and their intermediate steps, and still converge on the same incorrect final value. The measurement that separates these cases is the conditional error rate: given that one sample is wrong, how often is another sample wrong with the identical answer?

The conditioning is the part that carries the design, so it is worth saying why the unconditional version would be useless. Ask simply how often two samples agree and the answer is dominated by the cases where both of them are right, because on a task running in the eighties most pairs are two correct answers agreeing with each other. That figure rises when the model gets better and says nothing about whether a fan-out will help. Restricting attention to pairs where both samples are wrong removes the correct answer from the picture entirely and leaves only the question the vote depends on: when the samples fail, do they fail onto the same value or onto different ones. The function below computes it by walking every pair inside each case, counting the pairs where neither member matches the label, and among those counting the pairs that match each other.

import itertools

def error_correlation(samples: dict[str, list[str]], gold: dict[str, str]) -> dict[str, float]:
    """How often do two wrong samples agree with each other?"""
    pairs, both_wrong, same_wrong = 0, 0, 0
    for case_id, draws in samples.items():
        truth = gold[case_id]
        for a, b in itertools.combinations(draws, 2):
            pairs += 1
            if a != truth and b != truth:
                both_wrong += 1
                if a == b:
                    same_wrong += 1
    return {
        "p_both_wrong": both_wrong / pairs,
        "p_same_error_given_both_wrong": same_wrong / max(both_wrong, 1),
        "coverage": sum(
            any(d == gold[cid] for d in draws) for cid, draws in samples.items()
        ) / len(samples),
    }

Under errors that are independent across a reasonably large answer space, p_same_error_given_both_wrong should be low — two independently wrong extractions of an invoice total have little reason to land on the same wrong number. On the extraction task at temperature 0.7 it came out at 0.34 overall, and at 0.81 on the subset of invoices with a foreign-currency line, which is a slice where the model consistently applied the wrong conversion convention. That single number does more to predict whether N will pay than any amount of staring at accuracy curves, and it is available before you have run a single vote.

Translating 0.81 back into what the vote actually sees explains why that slice was hopeless. Roughly four wrong pairs in five carry the identical wrong number, which means the wrong answers on those invoices are not scattering across the space of possible totals — they are collecting on one value, the one the mistaken conversion convention produces. A vote needs the correct answer to hold a bigger pile than any single wrong answer, and a consistently misapplied convention builds precisely the pile that beats it. At 0.34 the wrong answers split often enough that the correct one usually survives the count; at 0.81 they have merged into a rival candidate. Same fan-out, same temperature, same code, and two slices sitting on opposite sides of the condition the method needs.

Coverage is the other quantity to look at, for a blunter reason. If coverage at N=5 is 84.5% against a per-sample accuracy of 84.0%, then in nearly every case where the model is wrong it is wrong in all five samples, and the ceiling on every aggregation strategy you might build — voting, judging, verification, reranking — is half a point away. The correct response to that measurement is to stop working on aggregation and go fix the prompt or the model, since no selection rule can select an answer that was never generated.

The half point is 84.5% minus 84.0%, and the reason it caps everything downstream is that every rule in that list picks one of the N strings that came back. None of them writes a new answer. So the best conceivable selector — one that identifies a correct sample whenever a correct sample exists — scores exactly coverage, which makes coverage the figure a proposal should be compared against rather than per-sample accuracy. A design review arguing about voting against judging when those two numbers are half a point apart is arguing about how to divide something that is not there.

The other knobs, and what each one costs#

When temperature alone leaves the failures correlated, the remaining moves all work by changing something the samples share rather than something they draw.

PARAPHRASES = [
    "Extract the fields from this invoice.",
    "Read the invoice below and report the requested values.",
    "You are processing an accounts-payable document. Return the fields listed.",
]

async def diverse_samples(doc: str, n: int = 5) -> list[dict]:
    """Vary the prompt and the example order, not only the sampler."""
    jobs = []
    for i in range(n):
        prompt = PARAPHRASES[i % len(PARAPHRASES)]
        shots = rotate(FEW_SHOT_EXAMPLES, i)      # same examples, different order
        jobs.append(call(
            system=prompt,
            messages=shots + [{"role": "user", "content": doc}],
            temperature=0.7,
            seed=1000 + i,
        ))
    return await gather(*jobs)

The seed=1000 + i in that code deserves a second look, because it appears to be a source of variation and is not one. A seed selects which draw you get out of a distribution; it does not change the distribution. Five different seeds at temperature zero return five identical answers, and five different seeds at temperature 0.7 return the same spread of answers the call would have produced with no seed set at all. What a seed buys is reproducibility — the ability to re-run a fan-out and get the same five samples back — which is worth a great deal when debugging and nothing at all for decorrelation. The two lines above it are the ones doing the work, because they change the system prompt and the example order, and those are inputs every sample would otherwise have held in common.

The paraphrase route most often earns its keep, because the failures that survive temperature are usually failures of interpretation, and a differently worded instruction is a different interpretation. It also has a diagnostic side effect worth taking advantage of: if one paraphrase disagrees with the other four consistently across a slice, you have found an ambiguity in the instructions, which is information you would otherwise have paid a human to discover by reading errors.

What lets that route reach further than the sampler is the one component every sample has in common. Temperature acts inside a single call, on tokens being drawn from a distribution the prompt has already fixed, so it can only ever vary what the samples draw. A paraphrase acts on the prompt itself, which is the component all N calls share, making it the only entry in the table that moves what the samples hold in common rather than what they draw. That is also the source of the risk in its last column. Changing what the samples share means the five calls are no longer answering an identical question, so if one paraphrase is materially weaker than the others, the vote is now averaging over instructions of unequal quality as well as over draws.

Mixing families is the strongest and most awkward option#

Errors rooted in one model's pretraining genuinely do not transfer, so a second family drops p_same_error_given_both_wrong sharply — on the same invoice set, adding one took it from 0.34 to 0.11. Against that, you are now maintaining two prompt variants, two sets of formatting quirks and two rate cards, and if one family is materially weaker, its samples dilute the vote rather than informing it. A vote of three strong samples and two weak ones is not obviously better than three strong samples, and it costs 67% more.

What moves that number when temperature could not is the ingredient a fan-out from one model is missing. Two families were trained on different corpora, with different tokenisers and different training runs, so a misconception one of them picked up has no particular reason to be present in the other. That is the same property classical ensembles obtain by resampling their training data, arriving here by a different route. It also explains why the drop shows up in the correlation figure rather than in per-sample accuracy: the second family is not more accurate than the first, it is differently wrong, and differently wrong is the whole requirement.

The 67% is worth checking rather than taking on trust, because it compares two designs and not two rate cards. The alternative being priced is three strong samples, so the base is three calls. The proposal is those three plus two weak ones, which is five calls. Two extra calls on a base of three is two thirds more spend, and what the ratio poses is whether two samples that are wrong more often than the others are worth two thirds of the bill for the chance that they break a tie.

Sampling for aggregation is not retrying#

Sampling N times for aggregation is a different operation from retrying on failure, even though both produce multiple calls. A retry is conditional and sequential: something failed a check, so you go again, and you keep the second answer because the first was invalid. Voting is unconditional and parallel, and it keeps a function of all N answers. The distinction matters operationally because retries are cheap in expectation — most requests do not retry — and voting costs N× on every request without exception.

Setting the two cost shapes side by side shows how far apart they are. A retry policy pays for one call always, plus a second call on whatever fraction of requests failed the check, so its expected cost is one call plus that failure rate times a call — on a service that fails a check a few percent of the time, a few percent above single calling. A fan-out pays N calls on every request, including the large majority that a single call would have answered correctly. The two land in completely different places on a bill even though both are described as calling the model more than once, and the entire difference is that one of them is conditional on a signal and the other is not.

Sampling for aggregation is also not tree search. Sampling five flat chains and counting the endpoints does not explore a space in any structured way, and it never lets a promising partial path spawn children while an unpromising one is dropped. Methods that do that exist and can outperform flat sampling on problems with a real search structure, at considerably more orchestration complexity. Flat sampling with an aggregation rule stays popular because it is trivially parallel and requires nothing from the model beyond running it again.

Two numbers, then, before N is chosen and before any aggregation rule is written. The distinct-answer count, which says whether the samples differ at all and costs nothing to log. And the probability that two wrong samples carry the identical wrong answer, which says whether the differences reach the part that can be wrong, and which is computed from one collection of samples against labels you already have. A high distinct-answer count with a high error correlation is the configuration that looks healthiest and pays least: the samples are visibly varied, the vote share is reassuringly short of unanimous, and the wrong answers still win.

Everything so far has assumed the answers are things you can put in a Counter. On an extraction task that holds, since each field is a string or a number that either matches or does not. The moment the output is a paragraph, the vote has nothing to count, and the technique turns into something else with a different cost structure and a different set of biases.