Self-consistency and shared errors

Self-consistency is the practice of running the same prompt through the model several times instead of once, letting the samples come back different from one another, and returning whichever answer the largest number of them agree on; the counting step at the end is the majority vote, and N is however many samples you decided to pay for. It exists because one call to a language model is a single draw from a distribution over possible answers, so a prompt the model usually handles correctly can still come back wrong on the particular draw you got, and asking several times lets those stray draws be outvoted by the ones that landed where the model normally lands. You meet it as an n or samples field in an eval config, as a fan-out of concurrent requests in service code, or as a vendor feature sold under a name like consensus mode, and it is usually the first thing proposed when a model-backed service has to become more accurate and retraining is not available. Its close relative, best-of-N, keeps the same fan-out of N samples and replaces the counting with a selection step, so that a scoring function or a second model picks one candidate out of the N.

The reason to understand what the vote estimates, rather than treating it as a general-purpose accuracy knob, is that the technique has a ceiling set before you choose N, and the ceiling is different on every slice of your traffic. Averaged across a whole stream the result usually looks like a small win. On the slice that prompted the project, it can be nothing at all, and the shape of the failure is the same one that makes the aggregate look fine.

What the vote estimates#

Sampling the same prompt N times and taking the modal answer is an estimator, and what it estimates is the mode of the model's answer distribution for that input. Sampling noise moves individual draws away from that mode; averaging over draws moves you back towards it. As N grows the vote converges to the single most probable answer under the model, which is approximately the answer greedy decoding would have given you for free. So the whole benefit lives in the gap between "one draw from the distribution" and "the peak of the distribution", and the size of that gap is set by how spread out the distribution is.

The convergence is worth doing slowly, because it is the one piece of machinery the whole technique rests on. Each call is a fresh draw from the same distribution, and the calls cannot see one another — nothing in the second request tells it what the first returned. Over many such draws the share of the votes an answer collects settles at its probability under the model, so an answer the model would produce three times in ten collects close to three tenths of the votes once N is large. Whichever answer carries the largest probability therefore ends up with the largest share, and "most frequent among the N samples" and "most probable under the model" become the same statement. That is the entire argument for counting. It is also the argument's boundary: the draws are independent of one another, which is not the same property as the errors being independent of one another, and the distance between those two sentences is what the rest of this chapter is about.

The claim that the limit is roughly what greedy decoding returns needs one condition attached, because greedy decoding and the mode of the answer distribution are not the same object. Greedy decoding takes the highest-scoring token at each position in turn, and a sequence assembled from locally best tokens is not always the most probable sequence overall — an opening token that is slightly less likely can lead into a continuation that is much more likely. How far the two can drift apart depends on how many tokens are being chosen. A one-word label leaves almost no room for them to differ; a long chain of reasoning leaves a great deal. On a short classification output the vote really is converging on something greedy decoding would have handed over for nothing, and on a long generated derivation that equivalence is an approximation rather than a fact.

The gap between one draw and the peak splits in two, and the split is worth stating precisely, because it is what tells you in advance whether N will pay. Some of the error is dispersion: the distribution has mass on several answers, and a single draw sometimes lands on a wrong one even though the peak is right. Some of the error is displacement: the peak itself is on the wrong answer. Majority voting attacks dispersion and is completely inert against displacement. Push N high enough and you eliminate the first entirely and are left with exactly the second, which is a hard ceiling on what any amount of sampling can buy on a given task.

This is the same relationship you would recognise from averaging repeated measurements of a length with a slightly bent ruler. More measurements tighten your estimate of what the ruler says and bring you no closer to the true length, because the bend is in every measurement. Where the analogy breaks is in the direction that matters practically: with a ruler you can measure a known reference and discover the bend. With a model you have no equivalent, which is why displacement error is usually found by reading failures rather than by any statistic computed over the samples themselves.

The split is measurable on labelled data even though it is not visible from the samples alone. Run N samples per case on a held-out set and separate the cases where the modal answer is correct from the cases where it is not. The first group's single-sample error rate is dispersion — draws that missed a peak sitting in the right place, and exactly the error voting removes. The second group is displacement, and its size is the floor that no N will get under. A task where those groups split ten points to two has real headroom for sampling; one where they split two points to ten has almost none, and the sensible response is to stop working on aggregation and go read the second group's failures. That measurement costs one collection of samples and no design work, and it should precede the argument about N rather than following it.

The tempting inference, and the one that gets N chosen before that measurement is taken, runs the other way: a slice with low single-sample accuracy has the most room to improve, so it is the slice voting should help most. The premise is true and the conclusion does not follow from it, because room to improve and room voting has to work in are different quantities. Voting can only recover error that varies between draws. A slice can be wrong on most of its cases and have almost none of that error be the varying kind, which is exactly the shape of the worst slice in the table further down. Low accuracy says how much is broken. The dispersion–displacement split says how much of what is broken is the kind sampling can reach.

The regime the technique was designed for#

Where voting helps, the model's distribution is diffuse and correct-leaning. A support ticket reading "the tracking page says delivered but nothing arrived, and I already paid the restocking fee" comes back as order_status three times, shipping_issue once and billing once. No single draw is reliable, the peak of the distribution sits on the right answer, and five draws find the peak more often than one draw does.

Those five draws are worth counting out, because the whole mechanism is visible at that size. order_status holds three of the five votes and every other label holds one, so the mode is order_status and that is what the vote returns. A single call on the same ticket would have returned order_status three times out of five and something else the other two, so one draw is wrong here two times in five. The vote is wrong only when the draws fail to come out in that three-to-one-to-one pattern at all. Voting has not made the model better at this ticket; the distribution it is drawing from is unchanged. It has stopped the two unlucky draws from being the answer that ships.

The original result, in Self-Consistency Improves Chain of Thought Reasoning in Language Models, came from multi-step arithmetic word problems, where a chain of reasoning has a dozen places to make an independent slip and a slip in one chain has no reason to appear in the next. Each chain fails in its own way, the failures scatter across the answer space, and the correct answer accumulates the plurality — more votes than any other single answer, even when it is short of half the votes cast. The published gains were large, on the order of seventeen points on grade-school word problems, moving a chain-of-thought baseline from the mid-fifties to the mid-seventies, and the size of them is a large part of why the technique spread.

The word carrying the weight in that description is plurality, and it is a weaker condition than a majority, which is why the technique can work on tasks where most individual chains come out wrong. A vote does not need the correct answer to be backed by more than half the samples. It needs the correct answer to be backed by more samples than any single wrong answer is. When a chain slips on an arithmetic step, the number it finishes on depends on which step slipped and in which direction, so two broken chains have no particular reason to land on the same number. The wrong answers spread across many distinct values, each collecting a vote or two, while every chain that gets the arithmetic right lands on the same value. The right answer has only to beat the largest of the wrong piles, and when the wrong piles are all of size one that is a low bar to clear.

It is worth noticing what the word-problem task had going for it. The answer was a number, so two chains either agreed exactly or did not. The reasoning was long, so there were many independent places to go wrong. And the errors were mostly arithmetic slips and dropped constraints rather than misunderstandings of the question, which meant the wrong answers scattered across a wide space while the right answer was a single point draws kept returning to. A classifier with eleven labels and a two-sentence input shares the discrete answer space and almost none of the rest, so importing seventeen points as an expectation was never reasonable.

The scattering is the property that does not survive the move to a classifier, and the reason is structural rather than a matter of degree. With eleven labels there are only ten wrong places for a sample to land, so wrong answers cannot spread thin — they pile up on whichever wrong label the prompt makes attractive, and a pile is exactly what a plurality has to beat. The short input compounds it. There is no long chain to slip in: the model reads two sentences and emits a label, so the places where an independent slip could occur have mostly been removed from the task before sampling starts. Both of the conditions that made the word problems work are weakened at once, which is how a technique gets reported honestly at seventeen points and delivers two.

Agreement is a per-slice property#

The measurement that separates the two regimes is agreement against correctness, computed within slices rather than over a whole stream.

from collections import Counter

def vote(samples: list[str]) -> tuple[str, float]:
    """Modal label and the fraction of samples backing it."""
    counts = Counter(samples)
    label, n = counts.most_common(1)[0]
    return label, n / len(samples)

def slice_report(cases, sample_labels, gold):
    unanimous = 0
    unanimous_correct = 0
    for case in cases:
        label, share = vote(sample_labels[case.id])
        if share == 1.0:
            unanimous += 1
            unanimous_correct += (label == gold[case.id])
    return {
        "unanimity_rate": unanimous / len(cases),
        "accuracy_when_unanimous": unanimous_correct / max(unanimous, 1),
    }

Run across a held-out set by slice, that produces the table a change of this kind should be reviewed against.

The fifth row has the highest unanimity of any slice and the lowest accuracy among the unanimous cases. Everywhere else, five-out-of-five agreement is a strong indicator that the answer is right; on that slice it indicates nothing, because agreement and correctness have come apart. Ninety-one percent of the time the five samples return the same label, and slightly under half of those times the label is wrong. Voting cannot repair that, since the vote's only job is to find the label the model was already going to produce, and the model was going to produce this one with conviction.

Reading that row back as a distribution shape makes the inertness concrete. Unanimity at five out of five means the distribution has nearly all of its mass on one label. Accuracy of just over half among those unanimous cases means the label carrying that mass is the wrong one almost as often as it is the right one. Put together, the slice holds a sharp peak sitting on the wrong answer about half the time, and a sharp peak on a wrong answer is the definition of displacement. Counting draws cannot move a peak, because counting draws is how you find out where the peak already is. Raising N on this slice buys a more precisely measured location for a peak whose location was never in doubt.

An error of that shape does not vary across samples for the same reason that a compiler does not vary across compilations. Temperature perturbs the token-level distribution; it does not introduce a second reading of the instructions. Once the prompt has committed the model to a particular interpretation, every sample inherits that interpretation, and the draws differ only in wording that has no bearing on the answer.

The steps under that sentence are worth having, because the conclusion is only as strong as the weakest of them. The model produces a score for every candidate next token, and temperature acts on those scores, so the only thing temperature can vary is which token gets drawn at a position where a token is being drawn at all. The prompt is not drawn — it is supplied — so every call reads identical instructions, and whatever those instructions determine is determined identically in all N calls. An ambiguous queue description is part of the prompt. The reading it produces is therefore fixed before sampling begins, and sampling is the only thing the five calls do differently.

The condition attached to that argument is output length, and it is the condition that explains why one technique behaves so differently on two tasks. Where the model emits a long chain of reasoning, part of its reading of the problem is being re-derived inside tokens that are drawn, and temperature reaches it — a chain can take a different turn at a sampled word and arrive somewhere else entirely. Where the model emits a label and stops, there is almost nothing drawn for temperature to act on, and the reading is carried entirely by the fixed prompt. A slice whose failure lives in the reading is unreachable by sampling in exactly the proportion that the reading lives outside the sampled tokens.

Five samples are not an ensemble#

A common way this gets misread is treating self-consistency as an ensemble. Ensembles in classical machine learning — bagging, random forests — get their gains from models trained on different data, so their errors genuinely decorrelate, and the ensemble's bias can be lower than any member's. Five samples from one model on one prompt are not that. They are five draws from a single distribution, so their errors are correlated by construction, and the ceiling is the mode of that distribution rather than anything better than it. The name "self-consistency" is accurate about this: the method is checking the model against itself, and a consistent model with a wrong belief is exactly what it will fail on.

Setting the two side by side shows which ingredient is missing. A bagged ensemble trains each member on a different resample of the data, so a quirk one member picked up from its particular subset is simply not present in the next member's training, and averaging the members cancels it. The cancellation depends entirely on the members having been built from different information. Five samples of one prompt share the weights, the prompt, the tokeniser and the decoding rules, and the only thing that differs between them is which token the sampler happened to draw. Anything determined by the shared parts is therefore common to all five by construction, and a misreading of an ambiguous instruction is determined by the shared parts. There is no resample here for a quirk to be absent from, which is why errors that are correlated here are correlated by construction rather than by bad luck.

There is a related mistake on the diagnostic side. Because voting helped on five slices out of six, the natural next step is to look at the aggregate unanimity rate — 78% across the whole set above — and conclude the model is well calibrated. The aggregate hides the inversion. The one slice where agreement did not track correctness is invisible in the pooled number, and it is precisely the slice where anybody would want a confidence signal, since it is the slice generating the escalations. Any use of agreement as confidence has to be validated per slice before it is trusted.

What makes that aggregate reading tempting is that almost any spot check confirms it. Five of the six slices in the table have agreement tracking correctness honestly, and they carry the large majority of the traffic, so a handful of cases pulled at random from the stream will nearly always come from a region where the signal works. The belief is not carelessness. It is a generalisation supported by most of the evidence anybody is likely to look at, which is the hardest kind to dislodge, and the only thing that dislodges it is computing the same two columns inside each slice instead of over the pool.

The inversion is not the only shape worth knowing about, and treating high agreement as automatically suspect is the overcorrection. A slice can have high unanimity and high accuracy — the first row of the table above is at 88% unanimous and 96.4% correct when unanimous — and on that slice agreement is doing exactly what it appears to do. What the table establishes is not that unanimity is meaningless but that its meaning is a property of the slice rather than of the method, so it has to be measured per slice and re-measured when the prompt or the model changes. A slice can also move between the two regimes without anything about the sampling changing: fix the ambiguous instruction and a displaced peak becomes a correct one, at which point unanimity on that slice starts carrying information it did not carry the week before.

The per-slice discipline is also what makes the aggregate gain readable. Two-point-two points across a whole stream is not a property of the technique; it is a weighted average of slices where it bought three points, slices where it bought nothing, and a base rate that decides how much of each you have. Quoting it as "self-consistency gives you two points" carries none of that, and it is the form the number travels in.

The first thing to establish about any proposal to sample N times is therefore not N and not the aggregation rule. It is which of your slices have a diffuse distribution around a correct peak and which have a sharp one around a wrong answer, because the first group is the entire budget the technique has to work with and the second group will consume spend and return unanimity. That is a measurement on labelled data you already have, and it is available before a single extra call is paid for.

Everything above also rests on one assumption stated plainly: that the N samples fail differently. Where they do, the vote extracts real information from the spread; where they do not, it spends N times the money to reproduce a single sample's answer with more confidence and no more accuracy. The recipe as usually written — sample five times at temperature 0.7 or 0.8, take the mode — contains nothing that establishes the assumption. It contains a knob that is widely believed to establish it, and that belief is where most of the disappointment with this technique comes from. The knob is temperature, and the thing it is believed to establish is diversity — which is a claim about how much the samples differ, not about whether their errors do. Whether the samples decorrelated, meaning whether the failures moved rather than only the wording, is a separate question with a separate measurement, and it is available from the same collection of samples the split above already needs.