Three sampled drafts can beat one draft criticised twice

The objection came from a reviewer who had read the design document carefully enough to quote chapter 2 back at the team. Their version: you are paying for two calls per review to have a model check work produced by the same weights, on the same training data, with the same failure modes, and you have already documented that the second call shares the first's blind spots. Spend the same budget on three drafts at temperature 1.0 and a cheap selector. You get three independent draws instead of one draw plus an expensive argument with itself, and you get them in parallel.

It is the strongest objection to this pattern and it deserves its strongest form before anything is said against it.

The case for deleting the critic#

Sampling has three things going for it, and none of them are rhetorical. The first is the arithmetic of independent draws. If a defect appears in a draft with probability 0.3 and the draws are independent, all three drafts carry it with probability 0.027, so a selector that can recognise a good draft when it sees one converts a 30% failure rate into something near 3% without any model needing to reason about anything. Nothing the critic loop does has that shape; a critic that catches 70% of a class caps out at 70%.

Work that number rather than accepting it, because the working is where the assumption lives. All three drafts carrying the defect means the first carries it and the second carries it and the third carries it. When the three events are independent, the probability of all three is the product of the three individual probabilities — three tenths times three tenths is nine hundredths, and nine hundredths times three tenths is twenty-seven thousandths, which is the 0.027. The word doing the work is independent: the multiplication is only valid if knowing the first draft carried the defect tells you nothing about the second. Hold on to that condition, because the second half of this chapter is about a class of defect where it is flatly false.

The second is that ranking is an easier problem than generating. A selector shown three drafts and asked which best satisfies the brief is doing comparison, where the alternatives supply their own contrast, and comparison is where model judgement is most reliable — which is the same reason the pairwise setup in Judging LLM-as-a-Judge holds up better than absolute scoring.

That claim is worth one step down, because it is the pillar of the objection that is easiest to nod at and hardest to state. Scoring a single draft on its own requires the model to supply a standard: it has to decide what a four out of five looks like before it can say whether this draft is one, and nothing in the input fixes that standard, so it drifts between calls and between drafts. Choosing among three drafts requires no standard at all. The alternatives supply the contrast, and the question collapses from "how good is this" to "which of these three better meets the brief", which is answerable by comparison against material that is present. That is why the selector can be a small, cheap call where a rubric grader cannot.

The third is latency, and it is the one that carries meetings. The critic loop is serial by construction: write, then criticise the thing you wrote, then revise it. Sampling is embarrassingly parallel.

The two totals in that diagram are its per-stage figures combined under two different rules, and the rules are the whole argument. On the serial side the stages happen one after another, so the total is their sum: write plus criticise plus revise is the 9.4 seconds on the label. On the parallel side the three drafts happen at once, so the three of them together cost what one costs, and the total is one draft plus the selector — the 5.6 seconds on the other label. Adding a fourth stage to the serial arm adds its full duration; adding a fourth draft to the parallel arm adds nothing at the median.

def best_of_n(brief, sources, n=3):
    with ThreadPoolExecutor(n) as pool:
        drafts = list(pool.map(lambda _: write(brief, sources, temperature=1.0),
                               range(n)))
    blocks = "\n\n".join(f"<draft id={i}>\n{d}\n</draft>"
                         for i, d in enumerate(drafts))
    r = client.messages.create(
        model=MODEL, max_tokens=300, system=SELECTOR_SYSTEM,
        messages=[{"role": "user",
                   "content": f"<brief>{brief}</brief>\n\n{blocks}\n\n"
                              "Return the id of the draft that best meets "
                              "the brief, and one sentence of reason."}],
    )
    return drafts[int(re.search(r"\d", r.content[0].text).group())]

Two things about that last line decide how far the code carries. All three drafts go into one call rather than one call each, which is what makes the selector a single call whatever n is, and it is why the call counts below read as four rather than six. And the chosen id is recovered by taking the first digit anywhere in the reply, so the design assumes the model puts the id ahead of the sentence of reason and assumes n stays in single digits. Both assumptions hold for three drafts and a compliant model, and both are the kind of thing that surfaces as a production incident rather than as a test failure.

Sampling with a selector is also older and better established than the critic loop, which is part of why the objection lands. Reranking several samples with a scoring model was standard practice in machine translation and speech recognition long before language models were general-purpose, and best-of-n against a reward model — a separate model trained on human preference judgements to score a candidate the way a human rater would, which is also the signal used in reinforcement learning from human feedback, or RLHF — is the technique that made the first generation of instruction-tuned assistants usable at inference time. A reviewer who has watched three cycles of elaborate generation schemes get beaten by a well-tuned baseline is applying an accurate prior, and the burden of evidence sits with the more complicated design.

That is four calls against the loop's three, but the selector's output is 300 tokens rather than 900, so the token bill lands at 5,200 against 4,100 — 27% more spend for 40% less latency, which on a quarter-end batch of 9,000 reviews is a real trade rather than a rounding error.

Both of those percentages are one division and worth confirming, since the trade is stated in them. The token ratio is 5,200 over 4,100, which is a little over one and a quarter — the 27%. The latency ratio is 5.6 over 9.4, which is a little under three fifths, so the saving is the remaining two fifths — the 40%. The reason the extra call costs so much less than a third more is in the sentence above it: the four calls are not four of the same thing, and a selector that reads three drafts and writes 300 tokens is a cheap call where a writer that emits 900 is not.

Where the argument breaks#

Independent draws only help with errors that vary between draws. Meridian's defect classes split cleanly on that property once someone bothered to check, and checking it took one script: generate three drafts per item, then count how many of the three carry each labelled defect.

The top rows are variance. A section goes missing because that particular sample wandered, and the other two draws have it, so a selector with a checklist finds a good draft immediately. The bottom rows are systematic: on 29 of the 34 pricing items, all three drafts stated the same wrong entitlement figure, because the figure comes from a prior over what SaaS plans include rather than from anything that varies with the sampling seed. Temperature moves the wording; it does not move the belief. Sampling a claim three times from a distribution centred on the wrong answer gives three wrong answers, and the selector, running the same weights, ranks all three as fine — which chapter 2 predicted and this table measures.

That table also gives a hard ceiling on what sampling can do per class, which is the number to quote when somebody proposes it. Sampling can only help on an item where at least one of the three draws is clean, so the reachable items are the ones in the first two columns. On constraint_violation that is six plus eleven, which is seventeen of the forty-three — so a perfect selector fixes at most seventeen and the other twenty-six are untouchable by any number of draws at this temperature. On contradicts_source it is two plus three, which is five of the thirty-four. Compare those ceilings against the separated critic's measured catches on the same classes and the argument stops being about architecture. The residual is that a ceiling is not a yield: reaching it needs a selector that reliably picks the clean draw, and the selector is the weakest component in the parallel arm.

The stumble worth naming here is the one the reviewer's phrase invites. Three drafts at temperature 1.0 look independent, because they are visibly different documents with different sentences in different orders, and difference is what a reader checks. What varies between them is the wording. What stays fixed is the distribution they are drawn from, and a claim about Scale-tier entitlements comes out of the centre of that distribution every time. So the property a reader can see varies freely while the property that decides the answer is constant, and the bottom two rows of that table are what constant looks like when it is measured.

Matching the budgets for this comparison takes a decision that should be made out loud, because calls and tokens do not match simultaneously. Three calls against four favours the loop; 4,100 tokens against 5,200 favours the loop as well; wall-clock strongly favours sampling. Meridian matched on tokens by trimming the selector's output allowance, reported all three figures beside the quality numbers, and accepted that no single budget definition is neutral. Anyone who reports only the definition their preferred arm wins on should be asked for the other two, and the disagreement usually dissolves once the three are on the same slide.

Two limits on the composition, and the order to build in#

Two things limit the composition. The selector shares the writer's weights, so it inherits everything chapter 6 said about correlated judgement — Meridian's selector picked the draft a human grader also preferred 71% of the time, against 33% for random, which is useful and is not a verifier. And best-of-n's benefit falls off quickly with n on prose tasks: moving from 3 to 6 drafts improved the mean grade by 0.03 and doubled the writer's spend, because after three draws the remaining defects are mostly the systematic ones that more draws cannot touch.

The two figures on the selector are worth reading together. Picking at random among three drafts agrees with a human a third of the time by construction, so 71% is more than twice chance and the selector is plainly doing something. It is also wrong on nearly three drafts in ten, and it is wrong in a way correlated with the writer, since it is the same weights ranking their own output. That is the difference between a selector and a verifier: a verifier's failures are structural and countable, and a selector's are the writer's own preferences applied twice.

The falloff has the same explanation as the ceilings above, and stating it saves anyone from trying n larger still. More draws only ever help on the items where a clean draw is possible at all, and the two systematic rows put most of their items outside that set. Within the reachable set, three draws already find a clean one most of the time, so the fourth, fifth and sixth draws are competing for a shrinking remainder while each costs a full writer call. That is why the curve flattens where it does rather than somewhere further out, and why the answer to a disappointing best-of-3 is a check on the systematic rows rather than a larger n.

Where the field is going is worth knowing even though it is out of scope here. On tasks that carry a real verifier — code with tests, SQL that either runs or does not, a proof a checker accepts — selection is exact and free of correlated judgement, and best-of-n scales far further because the thing choosing among draws is not a model at all. The nearest equivalent for a report-writing feature is the direction Meridian went by accident in chapter 6: move every class that admits a mechanical check to one, and reserve model calls for the classes that genuinely require judgement.

The order these chapters arrived in is close to the order to build in, and it is cheapest to follow it rather than to start at the end. Ground the critic before you shop for a second model, because a class failing for missing knowledge will fail on every model and the retrieval fix costs an afternoon. Close the critic's category list and put a severity threshold on it before you measure anything, since an unconstrained critic's numbers describe its verbosity. Then build the frozen task set with per-class labels, because every decision after that point — separated or reflective, one model or two, critic or sampling, ship or revert — is decided by a per-class breakdown and none of them are decided by an aggregate score.

Instrument the loop so the standing questions stay answerable: findings per draft above threshold, the drop rate from the quotation check, the fraction of reviews that skip revision entirely, and per-class defect counts from a labelled sample every few weeks. When a model version changes on either side, re-run the error table rather than assuming the correlation held. When a new defect class shows up in the incident log, add the category, add items to the set, and measure it before adding prompt text about it.

The one claim to keep out of the design document is the one that made all of this necessary. A separate critic gives you a reviewer with no stake in the draft, which is worth its call on judgement-shaped defects and worth nothing on defects rooted in what the model believes. Everything else — the second provider, the union of two arms, three draws and a selector — is a question about where a specific product's errors come from, and each of those has an answer that a week of labelling and one afternoon with the harness will produce.