Three seeds, three different winners, one plateau

Seed variance is the difference between two runs of the same search that differ only in the random seed recorded in the run config. It is not grader noise and it is not run-to-run wobble in a score: the prompts themselves come out different, because a different seed sends the search down a different lineage from round one onwards. Three runs of one search, identical in every other respect, produced winners of 640, 1,340 and 890 tokens sharing almost no wording, scoring 0.831, 0.828 and 0.835 on an 86-case holdout against a hand-written baseline's 0.800.

The obvious reading of that table is that the 0.835 is the best measured result and preferring anything else means overriding the evaluation with taste. The problem with it is that there is no measured difference between the three to be best at. The 0.7 points separating the highest from the lowest is smaller than the holdout can resolve, so choosing on it is choosing noise, and it costs something every time. What the three runs do carry is two things a single run cannot: evidence about whether the gain to baseline reproduces, and a list of the instructions that all three searches found independently.

How one integer in a config file produces three unrecognisable prompts is the mechanism to have straight before the scores mean anything. The seed drives every shuffle in the run, and the first thing it shuffles is which cases each candidate is scored on in round 1. Two seeds therefore ask round 1 two different questions. Because the spread of a mean over a small draw of cases is comparable to the real differences between early candidates, the two runs get two different answers about which candidates are worth keeping. Round 2 rewrites only the survivors, so the runs are now proposing from different parents. From there the divergence compounds rather than washing out — every round narrows the population further, and populations narrowed from different starting sets do not find their way back to each other. That is why seed variance shows up as different text and not merely as a different number.

What 0.7 points is worth on 86 cases#

Comparing two prompts on the same cases is a paired comparison, which is much tighter than comparing two independent means, so it deserves the correct arithmetic rather than a rule of thumb. What matters is the spread of the per-case differences, not the spread of the scores themselves. Between two winners from the same search the per-case differences are small and mostly zero, with a standard deviation around 0.21 on this task; between a winner and the hand-written baseline they are larger, around 0.18 but with a consistent sign on a subset of cases. The arithmetic below is a bootstrap, which draws a new sample of the same size from the per-case differences you already have, with replacement, several thousand times over, and takes the mean of each draw; the middle 95% of those means is the confidence interval, the range of true differences the observed data is consistent with.

Why pairing tightens a comparison is one substitution. Cases differ enormously in difficulty, and most of the spread in any set of scores is that difficulty rather than anything about the prompt — a case that is hard for one prompt is hard for the other. Subtract the two scores case by case and the difficulty subtracts out with them, because on that case it was the same number on both sides. What survives in each difference is only the part the choice of prompt caused. So the column to take a spread of is the differences, and comparing two independent means instead throws the cancellation away and ends up measuring case difficulty twice.

The bootstrap is doing something equally plain under an intimidating name. The question is how much the mean difference would move if a different 86 cases had landed in the holdout, and the only sample of cases anyone has is the one already in hand. So that sample is made to stand in for the population: drawing 86 differences from it with replacement — some cases turning up twice, some not at all — manufactures a set that could plausibly have been drawn instead, and the mean of each such set is one answer the experiment could have returned. Do that several thousand times and the middle of the resulting spread is the range the data supports. The replacement is the load-bearing part, since drawing 86 out of 86 without it returns the same set every time and answers nothing.

import numpy as np

def paired_bootstrap(a, b, n=20_000, seed=0):
    """a, b: per-case scores for two prompts on the SAME holdout cases."""
    rng = np.random.default_rng(seed)
    d = np.asarray(a) - np.asarray(b)
    idx = rng.integers(0, len(d), size=(n, len(d)))
    means = d[idx].mean(axis=1)
    return d.mean(), np.quantile(means, [0.025, 0.975])

print(paired_bootstrap(seed29, seed13))   # +0.007, CI [-0.038, +0.051]
print(paired_bootstrap(seed29, baseline)) # +0.035, CI [+0.001, +0.070]

The interval on the winner-versus-winner comparison is roughly fourteen times wider than the difference it is meant to measure, which means the ranking of the three is decided by which cases happened to land in the holdout. The winner-versus-baseline comparison is a different matter: the interval excludes zero, barely, so there is weak evidence that the search found something real worth about three points. Weak is the right word — the lower bound is 0.1 points, and a single mislabelled case could move it.

To make the 0.7-point gap resolvable you would need a much larger holdout. For a paired comparison at the conventional 5% significance and 80% power — a design that detects a real difference of that size in four runs out of five, and raises a false alarm in one run in twenty when there is nothing there — the required number of cases is (1.96 + 0.84)² × σ_d² / δ², which with σ_d = 0.21 and δ = 0.007 comes to about 7,000 cases. At ten minutes of expert attention per labelled case that is over a person-year of work, spent to answer a question whose answer would not change what you ship. The gap is not merely unproven; it is not worth proving.

That formula is two squarings and a division, and each part has a reason worth having. The constants come from the two error rates the design commits to: 1.96 is how many standard errors from zero a result has to sit before a 5% false-alarm rate is respected, and 0.84 is the extra distance that buys an 80% chance of clearing that bar when the effect is genuinely there. They add to a little under three, and squaring gives a little under eight. The rest is the ratio of noise to signal. σ_d of 0.21 divided by δ of 0.007 is thirty, which says the spread of the per-case differences is thirty times the difference being hunted, and squaring that gives nine hundred. Multiply the two and 7,000 falls out. The square in the denominator is the part to remember: halving the difference you want to resolve quadruples the cases, so a gap this small does not become affordable with more budget. Seven thousand cases at ten minutes of expert attention each is well over a thousand hours, which is what "a person-year" means when it is written out.

The cost of selecting on it anyway is subtle and compounds. Each time you compare candidates on the holdout and keep the best, the holdout has participated in selection, so the number it reports for the survivor becomes a little optimistic in exactly the way a tuning score is. Do it once and the effect is negligible. Do it monthly for a year, always keeping the highest of three or four, and the holdout has been used to choose twelve times, and its reported margins have drifted upward with no corresponding change in what users experience. The holdout is the one asset in this workflow that cannot be rebuilt in an afternoon — topic 3.5 puts the replacement cost in person-hours — and spending it to arbitrate differences it cannot see is the most expensive thing on offer.

The mechanism doing that is the same maximum-of-several-draws that inflates a tuning score, running at a much smaller count of draws and therefore much more slowly. Take the highest of three holdout numbers when all three prompts are worth the same, and the number you keep is above what any of them is worth, for the plain reason that a draw only gets selected when it came out high. One such pick moves a reported margin by a fraction of a point and nobody would notice. What makes it matter is that the picks accumulate against a set that never resets: the twelfth pick is made on the same 86 cases as the first, the drift runs in one direction only, and no later measurement can subtract it back out.

The objective is flat, and the winners are samples from the flat part#

The interesting content of a three-seed result is not the scores. It is that three searches, starting from the same seed prompt with the same budget and the same objective, arrive at texts with almost no wording in common and end up at the same height. That is what a plateau looks like from the inside. Over a wide region of prompt space the objective barely varies, so the search wanders until its budget runs out and reports wherever it happened to stop. There is no optimum to find, and the specific sentences in any individual winner are largely arbitrary.

That has a practical consequence that is easy to miss. If the plateau is wide, the parts of the winners that are not arbitrary are the parts that recur across seeds. Diffing the winners against each other is a cheap and unusually informative exercise, and it is less mechanical than it sounds, because three texts that share so little wording produce a line-based diff of two entirely deleted files and one entirely added one. What works is normalising first: ask a small model to extract each winner as a list of atomic imperative clauses, one rule per line, then cluster the clauses across the three lists by meaning rather than by string, so that "cite the clause number verbatim" and "when you reference a clause, reproduce its number exactly as printed" land together. The extraction costs a fraction of a cent per prompt and takes a few seconds, and it needs one guard: require every extracted clause to quote a span that actually appears in the source prompt, and discard the ones that do not, because a model asked for a tidy list of rules will readily invent plausible rules the prompt never contained. What comes out is an inventory you can read in a minute, with each rule marked by how many of the winners carry it.

The follow-up is to hand-write a fourth prompt: the baseline plus only the convergent instructions, in your own house style, at close to baseline length. Score it on the holdout the same way. That prompt has an author, reads like the rest of the codebase, and can be edited by anyone on the team at two in the morning — and it is not a criticism of the search, because the search is what surfaced the convergent instructions in the first place.

Once quality is a tie, the decision moves to axes that are measurable and that differ substantially between candidates: prompt length in tokens, the cost that length implies, p95 latency — the response time that the slowest one request in twenty exceeds — and whether a human can edit the thing under pressure. The editability axis is the one people leave out of the table because it feels unmeasurable, and it is the one that shows up in an incident: a prompt whose structure nobody on the team chose is a prompt nobody can safely modify when a customer reports that the assistant is doing something specific and wrong at 03:00.

What the spread does and does not license#

The three-seed spread is an argument against selecting on the last digit, and it is not an argument that the search found nothing. The gap to baseline is consistent across all three seeds and in the right direction each time, which is more convincing than any single run's margin, because three independent searches agreeing is a different kind of evidence than one search's number being large. Reporting that the winner reproduces across seeds — same rough magnitude, same convergent instructions — is more useful to a reviewer than reporting the best score.

Seed variance is also distinct from grader variance, and the two are often conflated. Grader variance is what you get by scoring the same prompt twice with a sampling judge; you remove it by pinning the judge's temperature or averaging several judge samples. Seed variance persists with a perfectly deterministic grader, because it comes from the search taking a different path, and it is the one that tells you about the shape of the objective. If you have never measured it, you do not know whether your reported gains are findings or draws.

The opposite outcome needs its own reading, and it is the one that quietly wastes money. If the three seeds come back with winners that are nearly identical — the same structure, the same rules, wording differing in a clause or two — the natural conclusion is that all three searches converged on something real. That is one of two explanations and it is the less common one. The other is that the seed never reached the part of the loop that would have made the runs differ, which happens easily: a framework that seeds Python's global random while leaving the proposer's sampling to the provider, or a proposer accidentally configured at temperature 0, will hand all three runs the same round-1 candidate pool and the same minibatch draws. The check belongs in the run logs rather than in the prompts. Record a hash of each round's minibatch indices and of each round's candidate texts, and compare those across the three runs: if round 1 differs and the lineages have visibly separated by round 3, then agreement at the end is evidence, and if round 1 is byte-identical you ran one search and paid for three.

A third outcome is possible and is the least discussed: three winners that beat the baseline by a consistent margin and share no instruction at all. The margin is then real and the mechanism is unreadable, because nothing in any single winner is corroborated by the other two. Hand-writing is not available in that case, since there is nothing convergent to hand-write, and the honest options are to ship one machine artefact with an owner and an annotation attached, or to treat the result as a finding about the objective rather than about the prompt and go looking for what all three were pulling on.

Three seeds is a floor rather than a target. Two runs cannot tell you which of them is the outlier, and five cost two and a half times as much to sharpen a distribution whose shape three runs already show. On this task each run took six hours and $47, so a sweep is $141 and one night if the runs go in parallel against separate rate-limit buckets, or most of two nights if they queue behind each other. The small amount of plumbing that makes them parallel is worth doing early, because a sweep that occupies three nights becomes a sweep of one seed the first week somebody needs an answer before Thursday.

The reporting rule that falls out of this is short. Run the search under at least three seeds. Report the range of holdout scores rather than the maximum, state whether the gap to baseline reproduces under every seed, list the instructions that appear in more than one winner, and name which candidate you are proposing and on which axis you chose it. A reviewer reading that can disagree with you about something real, which a single maximum does not permit.

There is one candidate in the table that has not been dealt with. Seed 13 is 2.4 times the baseline's length, adds 180 ms to p95, and contains — buried in the role-setting paragraph — an instruction telling the model to answer with complete confidence and never to state that it is unsure. It scored 0.828 on a clean holdout, which is to say the grader liked it. Deleting that sentence and shipping the rest is the obvious move and the wrong one.