Burn rate and transfer ratio

Advice about how long an eval set lasts arrives as a number: thirty iterations, or a quarter, or one release cycle. Those numbers are all real observations from somebody's project and none of them transfers, because the quantity they describe is a property of four things that differ between teams by more than an order of magnitude. A 40-case set under an aggressive week of prompt search and a 150-case set with graded scoring under six variants a week are not the same instrument being used at different speeds; they are two instruments with different lifetimes.

This chapter makes the lifetime measurable. It starts with what a single case is worth on a small set, which is where the arithmetic of selection becomes uncomfortable fastest; sets out the four factors that move the burn rate and which of them you can actually change; and then defines the transfer ratio, the share of a stretch of movement on the tuned-against set that also shows up on an independent one, which is the observable that turns all of this from folklore into a logged quantity.

What a point is worth on a small set#

Start with a 40-case eval set for a support-ticket router. Each case is worth 1/40 of the score, which is 2.5 points, so a move from 0.70 to 0.79 across a week of work is 3.6 cases changing their answer. Rounded to whole cases, four tickets that used to route wrong now route right, and every claim anybody makes about that week is a claim about four tickets.

Selection alone produces about that much. Suppose that of the 40 cases, 20 are answered correctly by every plausible variant, 8 are beyond the router regardless, and 12 sit on the boundary where the outcome depends on things no prompt edit controls. The number of borderline cases a given variant happens to get right has a standard deviation, the typical distance from the average, of about 1.73 cases, which on a 40-case denominator is 4.3 points. Taking the maximum over 25 variants sits about 1.97 standard errors above the mean, so the expected best-of-25 score is about 8.5 points above the true quality of every variant in the batch.

Each of those three figures is one operation, and they are the same three the simulation in the first chapter of this topic ran on a different set. The spread of a count of coin flips is the square root of the number of flips times a half times a half: twelve times a half times a half is three, and the square root of three is about 1.73 cases. Turning cases into points divides by the 40 cases in the set, giving 4.3 points — and that denominator is the whole reason this set is fragile, because those same 1.73 cases would be barely more than a point on a set of 160. The multiplier for the maximum of 25 draws comes off the same slowly-climbing curve as before, and just under two standard errors of 4.3 points is the 8.5.

Against an observed gain of 9.0 points, that leaves nothing over. The search manufactures 8.5 points from twenty-five genuinely identical prompts, and no arrangement of the numbers separates the manufactured part from a real one, because the same 40 cases both produced the improvement and measured it. This is not a claim that such a week is wasted — some of the edits probably do route tickets better — it is a claim that the set is incapable of saying which, and that the incapacity is arithmetic rather than a defect in the labelling.

The four factors#

Set size is the dominant one, because the standard error scales with the inverse square root of the number of cases. Going from 40 cases to 160 halves the noise, which halves the selection gain available at any search breadth. The inverse square root deserves one sentence of derivation, because it is the same law behind every sample estimate in this topic. Errors on individual cases are independent, so they combine as variances rather than as sizes, and a sum of n independent errors has a spread growing like the square root of n rather than like n. Dividing that sum by n to turn it into a rate leaves a spread falling like one over the square root of n. Four times the cases is therefore twice the precision — which is why 40 to 160 halves the noise, and why halving it once more would cost four times as many cases again. It is also the factor most under your control: 120 more labelled cases at the four-minutes-per-case rate from topic 3.4, on building a golden dataset, is about eight hours of work, and running a 160-case set costs roughly $1.30 and six minutes of wall clock per run at eight-way concurrency. For a team burning a 40-case set in a week, growing the set is the highest-leverage change available and it is cheaper than the week they just spent.

Search breadth is the second, and it enters logarithmically rather than linearly. Doubling from 5 variants to 10 adds about 0.4 standard errors of free gain; doubling again to 20 adds another 0.35. Each doubling adds roughly the same small amount, which is what growing with a logarithm means: the increments are constant per doubling rather than per variant. That is the good news and the bad news together, because it means restraint helps less than people expect — cutting a 30-variant week to 15 removes one doubling's worth of gain and no more, about a quarter of the selection bias. Breadth also compounds across weeks in a way single-week counts hide, since week two's variants are all descendants of a winner already fitted to the set.

How aggressively edits target observed failures matters more than either, and it is the factor that never appears in the arithmetic. An edit written from a general principle — "always resolve pronouns to the most recent named entity" — encodes no information about any particular case. An edit written after opening the three cases that failed and noticing they all involve a clause header inside a table cell, then adding "when the heading appears in a table cell, read across the row", encodes a great deal about those three cases. The second kind of edit is what makes prompt work productive and it is also close to a lookup table, so a week of nothing but failure-driven edits burns a set several times faster than a week of principled ones.

The reason this factor never turns up in the arithmetic is that it moves information through a channel the arithmetic does not model. A best-of-k calculation assumes each variant was written without reference to the set, so the only thing travelling back is the keep-or-discard bit. An edit written after opening three failing cases was not written that way: those case ids are inputs to it, and their content — a clause heading inside a table cell — is encoded directly in the instruction that comes out. The selection bit is still there and it has become the smaller of the two channels. A week of failure-driven work can therefore burn a set faster than any of these tables predict, which is why the tables should be read as a floor on the damage rather than an estimate of it.

The distinction between a principled edit and a failure-driven one is not that one is legitimate and the other is cheating; both are normal, and the second needs an independent check much sooner.

Score granularity is the fourth and the easiest to overlook. A binary correct-or-incorrect judgment on 40 cases has 41 possible values and a large per-case variance. A 0-to-4 rubric on the same 40 cases has finer resolution and lower variance, because partial credit is more stable than a threshold — a change that improves an answer from 2 to 3 registers, where a binary check would have called it wrong both times.

The mechanism is the threshold itself. Underneath any binary check there is a continuous quality, and the check reports only which side of a line the answer landed on. For a case sitting near that line, a difference too small to care about flips a whole unit of score, so the scoring rule is manufacturing variance out of nothing anybody wanted measured. A graded rubric moves in proportion instead, so less of the run-to-run movement is an artefact of where the line happened to sit. Since the free gain is the multiplier times that movement, shrinking the movement shrinks the gain by the same factor, and it costs a rubric rather than a labelling round.

Teams that move from binary to a graded rubric typically see their per-run standard error drop by 30% to 50% on the same cases, which cuts the selection gain by the same proportion for free.

Logging variants and checkpoints#

None of that gives you a number for your own team, and the way to get one is to record every variant and check the transfer ratio at intervals: 8 points of movement on the tuned-against set alongside 4 points on the independent one is a transfer of 0.5, and the missing 4 points are fit. The log is small and it has to be written at the time, because reconstructing which variants were tried six weeks later is not possible from version-control history alone — discarded variants leave no commits.

from dataclasses import dataclass, asdict
from datetime import date
import json, math

@dataclass(frozen=True)
class Variant:
    variant_id: str          # "v0031"
    logged: str              # ISO date
    parent: str              # variant it was edited from
    change: str              # one line describing the edit
    motivated_by: list[str]  # dev case ids read before writing it; [] if principled
    dev_score: float
    kept: bool

@dataclass(frozen=True)
class Checkpoint:
    logged: str
    variants_since_last: int
    dev_score: float
    holdout_score: float

def transfer(prev: Checkpoint, cur: Checkpoint, dev_se: float) -> float | None:
    """Fraction of the dev movement that showed up on held-out cases."""
    d_dev = cur.dev_score - prev.dev_score
    if d_dev < 3 * dev_se:
        return None   # dev barely moved; the ratio would be noise over noise
    return (cur.holdout_score - prev.holdout_score) / d_dev

def append(path: str, record: Variant | Checkpoint) -> None:
    with open(path, "a") as fh:
        fh.write(json.dumps({"kind": type(record).__name__, **asdict(record)}) + "\n")

Two details carry most of the value. The motivated_by field is the only record of how failure-driven the search was, and counting the fraction of kept variants with a non-empty list gives a rough leakage-intensity number for the campaign — a team at 80% failure-driven should expect much lower transfer than one at 20%. The guard inside transfer refuses to compute a ratio when the movement is smaller than three standard errors, because dividing a noisy small number by another noisy small number produces ratios of 4.0 and −2.3 that mean nothing and get quoted anyway. A ratio misbehaves whenever its denominator can sit near zero: a small denominator makes the quotient enormous, and a denominator that might have come out negative makes the quotient change sign. Requiring the dev movement to clear three standard errors is a crude way of insisting that the denominator is a real quantity rather than a wobble, and three is the number because at that distance the denominator's sign is not in doubt. Checkpoints should be spaced so the movement between them clears that guard, which on a 200-case set with a 2.5-point standard error means waiting for at least 8 points and for most teams lands every three to five weeks.

Three campaigns compared#

Here are three real-shaped campaigns run in the same quarter, each logging as above. The point is the spread rather than any individual figure.

The router burned through its set in a single week of work and the summariser was still transferring 71% of its movement after ninety variants. Anyone quoting "a set is good for about thirty iterations" would have been roughly right about one of these three and wrong by a factor of four about another. The difference is entirely explained by the four factors: the router had the smallest set, the widest search, binary scoring and the most failure-driven edits, and every one of those pushes in the same direction.

The four do not add, they multiply, and that is why the spread is a factor of four rather than a few points. The router's set is a quarter the size of the summariser's, which roughly doubles its per-run noise before anything else happens. Its search is four times as wide, which adds most of another standard error to the multiplier. Its binary scoring manufactures variance the rubric does not. And its edits carry case content into the prompt where the summariser's largely do not. Any one of those on its own would have made it burn faster. Together they make it a different instrument, which is the real argument against borrowing an iteration count from somebody else's project: the number is not wrong, it is attached to four values none of which you were told.

The summariser column also shows what a healthy campaign looks like. Transfer above about 0.7 means most of what you are measuring is real and the set is still doing its job. Between 0.3 and 0.7 the set is half instrument and half mirror, and it is time to plan a refresh. Below 0.3 the score has become a description of the set.

The log is a rear-view mirror#

Recording variants and transfer ratios does not slow the burn by a single point. It is diagnosis rather than prevention, and it tells you where you are after the fact — the router team would still have burned their set in that week; they would simply have known by the following Monday instead of finding out in a release post-mortem two months later.

The transfer ratio is also noisier than it looks. It is a quotient of two differences, each carrying its own standard error, so a single checkpoint pair gives a wide range even with the three-standard-error guard in place; a 0.41 could easily be a 0.25 or a 0.60. It becomes trustworthy over three or four checkpoints, where the trend matters more than any value. And the ratios do not transfer between projects or even between features in the same product, because the four factors differ, which means every team has to measure its own and cannot inherit a number from this table or any other.

There is also a failure the log cannot see. A checkpoint requires a read of the independent set, and reads are the resource that runs out. A team diligent about measuring transfer every three weeks spends about seventeen reads a year on measurement alone, before any release decision needs one. Measuring the burn consumes the same budget as causing it, and the only way through is to space checkpoints so that each one answers a question a release was going to ask anyway.

The useful summary is that burn rate is not a constant and was never going to be. It is the product of a noise level you set by choosing a set size and a scoring scheme, and a search intensity you set week by week, and the two multiply. A team that knows its own numbers can trade between them deliberately — label more cases, or search less widely, or move from binary scoring to a rubric — where a team working from a published iteration count is guessing which of the four factors it happens to resemble.

That collision between wanting to measure the burn and not wanting to cause it is what has to be resolved by policy rather than by arithmetic, along with the harder question of what to do with a set that has already gone.