Self-consistency sampling

A free-form answer is one whose value to the reader is the prose itself — a summary, an explanation, a drafted reply — rather than a single field a program switches on. Chapter 2's method does not apply to it, because there is no label token whose distribution is the distribution over answers. Chapter 3 ruled out the average over the generated text, which mostly measures phrasing and length. Chapter 4 ruled out reading a single trace at all as a check on whether a stated fact is real. What is left is to stop asking one generation to report on itself.

The move is to ask the model the same question several times, with sampling switched on, and look at whether the answers agree. If four of five drafts cite clause 4.2 and one cites 6.1, that split is information about the model's uncertainty which no single generation exposes, because the four-versus-one distribution over answers is exactly what one sample collapses. This chapter is about what that costs, how coarse the resulting score is, and how to tell whether you need it at all.

Why that move works is worth stating slowly, because everything else in the method is bookkeeping on top of it. A single generation is one draw from a distribution over answers: the model has some tendency to produce each candidate answer from this prompt, and the sampler realises exactly one of them. One draw carries no information about the shape it came from, because a draw that lands on an answer the model was near-certain of and a draw that lands on one it barely favoured are the same object — a string, with nothing attached saying how close the contest was. Drawing again is the only way to see that shape, since no part of a completion reports the tendency that produced it. What N samples estimate is that tendency, which exists whether or not anybody measures it and which one call collapses into a single realisation.

The shape of a self-consistency run#

Diagram

Four things happen and each is a decision you have to make explicitly. The fan-out needs a sampling temperature, and the number of samples sets the resolution of the score. The extraction step pulls out whatever the downstream system actually consumes — a clause number, say — because comparing whole prose answers for equality returns zero agreement on almost every case. That step is also where most of the engineering effort goes, and where most implementations quietly go wrong: an extractor that returns None on a malformed sample, and a scorer that drops those samples before counting, will report perfect agreement on a case where three of five answers were incoherent. The grouping step needs a definition of when two extracted answers count as the same, which is less obvious than it sounds. The threshold turns a fraction into a routing decision, and where it goes is a question chapter 7 takes up.

The arithmetic of that extraction bug is worth seeing, because it is a defect that makes the dashboard look better the worse the run went. The score is the largest agreeing group divided by a denominator, and dropping the unreadable samples removes them from the denominator while leaving the largest group untouched. Two readable samples that agree, out of five drawn, score two fifths when the denominator is the number of samples and a clean 1.0 when it is the number the extractor managed to read. The samples thrown away are precisely the ones that disagreed about what shape the answer even had, so discarding them removes the strongest available evidence of trouble, and the score rises exactly where it should have fallen.

The method has a name and a provenance. It came out of Wang and colleagues' 2022 work on self-consistency, where the goal was accuracy rather than confidence: sample several chain-of-thought reasoning paths, meaning several written-out lines of working, for an arithmetic problem, take the majority answer, and accuracy on grade-school word problems rises by double-digit percentage points over a single greedy decode. The confidence signal was a byproduct — if you are already counting votes to pick the winner, the size of the winning bloc is sitting there. The current state of the art extends it: Farquhar and colleagues' semantic-entropy work, published in Nature in 2024, clusters the samples by bidirectional entailment — two answers land in the same cluster when each one implies the other — rather than by string equality, and computes entropy over the meaning clusters, which detects the case where the model states the same wrong thing in five different phrasings less badly than a string-match vote does. Both rest on the same move of turning one generation into a distribution you can look at.

What N samples cost#

Take a reply-drafting prompt of 850 input tokens — system instructions, a customer ticket, two retrieved policy excerpts — producing about 120 output tokens. At $0.60 per million input tokens and $2.40 per million output, a single draft costs $0.00051 in input and $0.000288 in output, or $0.0008 all in.

Five samples do not cost five times that, provided you use an n parameter or prompt caching so the prompt is encoded once. The input is paid once at $0.00051 and the output five times at $0.00144, for $0.00195 — a 2.4× multiplier rather than 5×. Five genuinely independent calls with no cache would cost $0.004, the full 5×. Which of those two you are paying depends on your provider and your client code, and it is worth confirming on a billing report rather than assuming, because the difference decides whether the method fits inside a budget.

The multiplier is a property of the prompt's shape rather than of the method, and that is the part which does not transfer between call sites. Input and output are billed at different rates and only the output is repeated, so the total is one input charge plus N output charges, and how far that sits below N times a single call depends on how much of a single call was input to begin with. On this prompt the input dominates the bill, which is why five samples land nearer twice a single call than five times it. On a prompt with a two-line instruction and a long generation the input charge is a rounding error, the output charge is nearly the whole bill, and the multiplier climbs back toward N. Reading 2.4× off this call site and carrying it to a summarisation route with a two-line prompt is how a fan-out arrives over budget with its arithmetic apparently already done.

Latency behaves worse than the cost does, and this is the part teams underestimate. Fanning out in parallel means the request finishes when the slowest of the five finishes, which is a maximum over five draws rather than an average. For a single call with a median of 900 ms and a 95th percentile of 2,100 ms, the median of the maximum of five draws lands at roughly the 87th percentile of the single-call distribution, around 1,750 ms, and the 95th percentile of the maximum lands near the single call's 99th, around 3,400 ms. In other words, your new typical response time is close to your old bad one.

The 87th percentile in that sentence is not an estimate pulled from experience. It follows from one property of a maximum, and the property is worth having because it tells you how the number moves when you change the fan-out. The slowest of five calls comes in under some time only when all five of them do, and if the five are independent draws then that joint chance is the single-call chance multiplied by itself five times. The median of the maximum is the time at which that joint chance reaches a half, so each individual call has to beat that time far more often than half the time — the number which, raised to the fifth power, gives a half is a little under nine tenths. That is the whole derivation, and it also says what happens as the fan-out widens: the more draws you take, the further into a single call's tail the slowest of them has to sit.

The last two columns are the ones people forget. With five samples the agreement score can only take the values 0.2, 0.4, 0.6, 0.8 and 1.0, so thresholds at 0.75 and 0.85 are the same rule, and a dashboard reporting "mean confidence 0.84" is reporting the average of a five-point scale. If you plan to tune a threshold to two decimal places, you need an N that can express two decimal places, and that N is expensive.

Where that column of distinct values comes from is arithmetic rather than measurement, and seeing it makes the constraint predictable for any N. The largest agreeing group is a count of samples, so it is a whole number somewhere between one and N, and dividing a whole number in that range by N can land on N distinct fractions and no others. Add the case where the extractor reads none of the samples and the largest group has no members at all, and the five-sample row's 6 and the ten-sample row's 11 both follow. The smallest difference the score can express is therefore one over N, which at five samples is two tenths, and any two thresholds sitting inside one of those gaps select exactly the same set of cases.

The resolution problem has a second half that is easy to overlook: the score is not just coarse, it is noisy. Five samples are five draws from the model's distribution over answers, so the observed agreement is a binomial estimate of the underlying proportion. If the model's true tendency is to give the majority answer 80% of the time, the standard error of a five-sample estimate is sqrt(0.8 × 0.2 / 5) ≈ 0.18, the standard error being the typical distance between the fraction you observe and the underlying one you are trying to measure. Two cases where the model is equally sure will routinely score 1.0 and 0.6, and a threshold at 0.8 will route one of them to a human and auto-send the other for no reason except which side of the coin came up. At ten samples the standard error falls to 0.13, which is better and still large. This is a structural limit of the method rather than an implementation detail: agreement scores are legitimate as a coarse three-band signal — clear agreement, clear disagreement, and the muddy middle — and are not legitimate as a continuous score you sort a queue by.

The formula behind that limit is worth walking rather than copying, because its shape is what decides how many samples any of this needs. Inside the square root, the model's tendency to give the majority answer is multiplied by its tendency to give anything else: eight tenths against two tenths, which is sixteen hundredths. That product is the spread of a single draw, and it is at its largest when the model is evenly split and at its smallest when the model is nearly certain, which is the sense in which a contested case is harder to measure than a confident one. Dividing by the number of samples — five, here — leaves a bit over three hundredths, and taking the square root of that returns the 0.18.

The square root is the term that makes resolution expensive, and it explains why the sample column in the table climbs faster than the usefulness of what it buys. Because the sample count sits under a root, halving the typical error costs four times the samples rather than twice as many, which is why moving from five samples to ten takes the error from 0.18 to 0.13 rather than to half of 0.18. A score precise enough to sort a queue by needs neighbouring cases to be genuinely ordered, and reaching that from five samples is a multiple of the inference bill that no reply-drafting budget carries.

Which decisions are actually free-form#

Before paying for any of this, check whether the decision really is free-form, because a surprising number of apparently free-form outputs are label decisions wearing prose.

The test is what the downstream system branches on. If the consumer of the model's output is a human reading three sentences, the decision is free-form and you need agreement. If the consumer is code switching on a field — a clause number, a routing queue, an amount, a yes/no eligibility flag — then the decision is a label decision, even though the visible output is a paragraph, and the confidence question is about that field rather than about the prose. The prose is packaging.

The test works because a confidence score is always a statement about one specific quantity, and the quantity worth measuring is the one an error would travel through. Where code branches on a clause number, a draft that phrases the same clause three different ways is three correct answers, and a draft naming a different clause is one wrong answer, whatever the sentences around it look like. Measuring the prose in that situation spends inferences on variation that has no consequence downstream, and it dilutes the single distinction that does have one. Where a human is the consumer, the prose is the decision and there is no narrower field to fall back on, which is the case that genuinely needs agreement.

Where the conversion does not apply#

Two situations resist that conversion and genuinely need sampling. The first is when the answer space is open — a summary, an explanation, a diagnosis with no fixed vocabulary — so there is no closed set to classify into. The second is when the failure you care about is in the reasoning rather than in the conclusion, since a second call over the finished draft sees only the conclusion. Those are the cases where you pay for N inferences, and it is worth being able to say which of your routes is which.

Both limits come from one property of the cheap route, which is that a classification call reads a finished text and reports how that text distributes over a set of options you supplied. Where no such set exists, there is nothing for the probability to be a probability over, and a score computed against a set of options invented for the purpose measures the options rather than the answer. Where the set does exist but the fault lies in the working that produced the conclusion, the classifier is reading a conclusion that has already absorbed the fault, and it reports confidently on a text that is internally consistent with itself. Sampling reaches both cases because it re-runs the step that went wrong instead of inspecting what that step produced.

What the conversion buys where it does apply is a legitimate confidence score over the thing that matters — which clause the reply is standing on — for a small fraction of the cost. What it does not buy is any check on whether the clause is the right one, or whether the prose around it says something the clause does not support. It converts a free-form decision into a label decision and then applies the label-decision method honestly.

The latency figures also decide where sampling can live in a product. At a median of 1,750 ms a fan-out is invisible in an asynchronous system that drafts replies for later review, and it is not invisible in a chat interface where a user is watching a cursor. One workaround is to return the first sample immediately and compute agreement in the background, using the score to flag the reply for retrospective review rather than to gate it — which is a different product behaviour, not a cheaper version of the same one, and worth naming as such when someone proposes it as a compromise. The other is to accept that some surfaces cannot afford a confidence signal at all, and to spend the engineering effort on making the single answer better instead.

What makes the background variant a different product rather than a cheaper one is the order of events. A gating score is computed before the answer is shown and decides whether it is shown at all; a retrospective score is computed afterwards, by which time the reader has already acted on whatever the first sample said. Both produce the same column on the same dashboard, which is why the substitution is easy to wave through in a design review. What the substitution changes is who carries the error, moving it from a reviewer who never sends a doubtful reply to a customer who has already received one.

Agreement measures reproducibility. Five samples agreeing means the model reliably produces that answer from that prompt, which is a real and useful property: it tells you the answer is not an artefact of one unlucky draw from the tail. It is not evidence that the answer is correct, and the gap between those two statements is where the method's reputation gets damaged. A model that has confidently learned a wrong association will reproduce it on all five samples and score 1.0. Chapter 4's fabricated citation, sampled five times at a low temperature, comes back five times.

The inference from agreement to correctness is tempting for a reason worth naming, because the half of it that holds is what makes the other half convincing. Disagreement genuinely is evidence of doubt: a model that splits its samples across three answers does not have one answer it reliably produces, and that is a fact about the model worth acting on. The step that does not hold is the reverse one, treating the absence of disagreement as the absence of doubt, and it fails because a model can be wrong for reasons that are stable across draws. Sampling varies the draw while holding the weights, the prompt and the training distribution fixed, so any error living in those three is reproduced by the method rather than exposed by it.

There is also a failure that happens before any of this reasoning applies, and it is the most common way self-consistency ships broken. The run above has two settings that nothing in the diagram forces you to choose deliberately — what temperature the samples are drawn at, and what "agree" means when you compare them. Leave either one to inherit a default and the number you report has no definition at all.