Measuring a prompt change

Before the apparatus exists, prompt work is settled by reading samples, and reading samples cannot distinguish two prompts from ten draws of one. A sampled completion is one point from a distribution; two samples of the same prompt differ about as much as two samples of different prompts on any single input, so a side-by-side of five before-and-after pairs is a comparison of ten generations rather than of two prompts.

That claim rests on what a model does on one call, and the steps under it are worth having in full, because every number in this chapter inherits them. The model returns a score for every candidate next fragment of text given everything before it. A separate step of ordinary code then picks one candidate from those scores and appends it, and where that picking step is allowed to be random — which is the provider default — the completion you read is one draw. Two calls on the same prompt and the same input are therefore two draws from one distribution, and on an input where that distribution is broad they land differently often enough to matter.

Changing the prompt moves the distribution. On any single input it usually moves it by less than the distribution is already wide. The difference between one draw from the old prompt and one draw from the new one is then dominated by which points happened to come up rather than by the edit, which is what makes five pairs uninformative no matter how carefully they are read. The shift is real all the same, and the property that hides it on one input is the property that recovers it across many: the shift is systematic and the draws are independent, so scoring hundreds of cases lets the shift accumulate while the draws cancel. Every rule below exists to keep that accumulation attributable to the edit rather than to something else that moved.

The cost of settling prompt work by reading pairs is not only bad decisions. It is that the team has no way to tell a good decision from a bad one afterwards, so every argument about a prompt is settled by seniority or by whoever is most recently annoyed. What replaces it is about eight lines of code and four rules, and the rules matter more than the code.

Four things a claim of improvement requires#

The cases must be fixed and held out. Fixed means the same inputs for both prompts, ideally the same inputs for the next six months, so that a score in March is comparable with a score in January. Held out means they are not the cases you were reading while you wrote the prompt. The cases you tune against get fixed specifically — you will read a bad output, add a clause that handles it, and the clause will handle that input and nothing resembling it — which is why teams that iterate on twenty examples end up with a prompt that scores wonderfully on those twenty and moves not at all in production. Keep a development set you look at freely and an evaluation set you never read individual outputs from except when investigating a regression.

The decoding settings must be fixed. Temperature, top-p (the share of probability mass the sampler is allowed to draw the next token from), max tokens, model version and system prompt all move the score, and a comparison in which the new prompt was run at temperature 0.2 because someone was debugging is not a comparison. Pin the model to an exact version string rather than a floating alias; a provider rolling an alias forward under you is the single most common source of a regression nobody changed anything to cause.

The metric must be stated and applied identically to both. "Quality" is not a metric; what goes in the pull request is a sentence of the form the proportion of the 400 cases where the reply parses, the category is one of the six, and the category matches the human label. Metrics differ enormously in what they can see and what they cost:

The third column is the one that decides how a metric behaves for the rest of this chapter, and the entries in it have a cause rather than a convention. Exact match against a label reads one sampled decision, the model's, so its spread is borderline cases flipping and nothing else. A judge — a second model call that scores each reply against a written rubric — adds a second sampled decision on top of the first. The judge's verdict is itself a completion drawn from a distribution, in exactly the way the reply being judged was, so a case can move between runs because the answer changed, because the verdict changed, or because both did. Two sampled decisions in series is why a judge's spread is roughly double an exact-match metric's rather than equal to it.

Judge drift earns its own clause in that cell because it is a different failure from spread and repeats do not catch it. Spread is movement around a fixed centre, so averaging over runs converges on that centre. Drift is the centre itself moving — a provider updates the judge model, somebody edits the rubric — and every run after such a change agrees with every other run after it, which is precisely what makes repeats blind to it. Repeats are the defence against spread. Re-checking the judge against human labels on a fixed sample is the defence against drift, and it is why a stack that contains a judge also contains a periodic human pass.

Human review sits in the table for completeness and its uncertainty is not the same quantity as the other three. Two raters disagreeing on a tenth of the cases is not a score moving between runs. It is two instruments giving different readings of one case, and what that number bounds is how well the label itself has been defined.

Most production setups end up with a stack: schema validity as a hard gate that must be 100%, exact match on the subset of cases with a single right answer, and a rubric-scored judge for the open-ended remainder, with a human pass on 50 cases quarterly to check that the judge still agrees with people.

The fourth requirement is that the gap be larger than the noise, and it is the one that gets skipped even by teams who have the other three.

Measure the noise before you measure the change#

Run the unchanged prompt three times over the same fixed cases and record the three scores. On a 120-case classification set at temperature 0.7 with an exact-match metric, a typical result is 71.7, 72.5 and 70.8 — a spread of 1.7 points with nothing changed at all. Any prompt edit that moves the score by less than that has told you nothing, and a team that does not know this number will ship a change measuring +1.2 and then spend a morning explaining why it measured −0.4 the following week.

It is worth tracing where 1.7 points of movement can come from when the prompt, the cases, the model and the metric are all identical, because the answer decides how much you can borrow the number from anyone else. Most of the 120 cases produce a distribution sharp enough that one candidate holds nearly all the weight, and a random draw from a distribution like that returns the same label on every run. Those cases contribute nothing to the spread at all. A small number sit near a boundary, where the top two candidates score close together and the draw is close to a coin flip between them. Each of those flips independently on each run, and a handful of flipped cases out of 120 is a point or so of score.

So the spread is a property of how many borderline cases your particular set contains. A set full of genuinely ambiguous inputs produces a wide one; a set of clear-cut inputs produces almost none, which is why the third column of the table above gives a range for exact match rather than a figure. It cannot be looked up, and a spread somebody else measured on their evaluation set is not evidence about yours.

There is a second and larger source of uncertainty that the three-run spread does not capture. Those 120 cases are a sample of the inputs you care about, and a score of 72% on 120 cases carries a 95% confidence interval of roughly ±8 percentage points around the true rate. Run-to-run spread tells you whether the difference is real on these cases; the confidence interval tells you whether it will hold on the next 120. Increasing case count shrinks the second and barely touches the first, which is the practical argument for evaluation sets in the hundreds rather than the dozens: at 400 cases the interval is about ±4.4 points, and at 1,000 it is about ±2.8.

The way those three intervals move is the useful part of them, and it follows from one property of an average. Each case is a single pass-or-fail draw, and the score is the average of those draws. Averages of independent draws grow more stable as draws are added, and the stabilising goes as the square root of how many there are rather than in proportion to them. Doubling the case count therefore does not halve the interval. It divides it by the square root of two.

Run that on the figures above. Four hundred cases is a bit over three times a hundred and twenty, and the square root of a bit over three is a bit under two, so an interval of roughly ±8 points divides down to about ±4.4. A thousand cases is a bit over eight times a hundred and twenty, and the square root of that is a bit under three, which takes ±8 to about ±2.8. Reading those two together gives the exchange rate: getting from eight points of interval to under three costs eight times the labelling, which is why evaluation sets stop growing somewhere in the hundreds rather than continuing to the tens of thousands.

Holding the two quantities apart is what stops a team labelling more cases in the hope of a quieter nightly run. Run-to-run spread asks whether a difference is real on the cases you already have, and it is set by how many of those cases sit near a boundary. The confidence interval asks whether the difference will survive on the next batch drawn from the same population, and it is set by how many cases there are. Adding cases moves the second directly. It moves the first only if the new cases happen to be less borderline than the old ones, which is not something you get to choose.

The harness that produces a score and a spread is not a platform. It is this:

def score(prompt: str, cases: list[Case], model: str, seed: int) -> float:
    hits = 0
    for c in cases:
        out = client.complete(model=model, system=prompt, user=c.input,
                              temperature=0.7, max_tokens=64, seed=seed)
        hits += int(normalise(out.text) == c.label)
    return 100 * hits / len(cases)

BASE = "claude-sonnet-4-5-20250929"
for seed in (11, 12, 13):
    print(seed, score(OLD_PROMPT, CASES, BASE, seed),
                score(NEW_PROMPT, CASES, BASE, seed))

Three seeds — a seed being the number that pins the sampler's random draw, so that a run can be repeated and two runs can be told apart deliberately — both prompts, one number each. Everything else, from dashboards to regression tracking to per-slice breakdowns, is an elaboration of those eight lines, and the elaboration is worth building only after the eight lines are being run.

One change per measurement, and what to do when you did not#

Suppose a commit changes four things in a prompt: it tightens the role line, adds a constraint about hedging, swaps one of the three examples for a newer one, and reorders the sections. The nightly eval falls 12 points. The eval set and decoding are fixed, repeat runs of the previous prompt varied by under a point, and the author is confident that all four changes were improvements.

Revert to the scored version first, because the production system is currently 12 points worse and the diagnosis is not urgent in the way the regression is. Then re-apply the changes one at a time against the same fixed set. On a real instance of this, the four changes measured like this:

Add the delta column up before reading anything into it, taking the four changes in the order the table lists them. The role line is plus three tenths. The hedging constraint is minus six tenths, which brings the running total to minus three tenths. The swapped example is minus nine point four, taking it to minus nine point seven. The section reorder is minus two tenths, and the total lands at minus nine point nine. Adding them like that is legitimate only because every one of the four was measured against the same reverted baseline on the same fixed cases, so they are four differences from one common reference rather than four steps along a path — had each been measured on top of the previous one, the column would be a running total already and adding it would double-count.

The individual deltas sum to −9.9 and the combined change measured −12.0, so about two points came from an interaction between changes rather than from any of them alone. The swapped example was the obvious culprit before the interaction was accounted for, and the interaction points at the pair the previous chapter would predict: the newer example, taken from a recent production transcript, hedged in exactly the way the newly added constraint forbade. The constraint alone cost half a point. The example alone cost nine. Together they cost more than the sum, because the demonstration was now contradicting a rule that had not previously existed to be contradicted.

The residue is worth stating as a subtraction. Minus twelve against a sum of minus nine point nine leaves two and a tenth points belonging to no single change, and an interaction is not a mysterious quantity — it is the part of the damage that exists only when two of the changes are present at once. Neither change carried a contradiction on its own: measured alone, the example sat in a prompt with no rule against hedging, and measured alone, the constraint sat in a prompt whose examples all obeyed it. The contradiction was manufactured by putting them together, which is the failure the previous chapter measured as a stated constraint's compliance falling by thirty-five points.

What follows from that is a limit on what bisection can tell you. A sum of one-at-a-time deltas is a lower bound on what a combined change will do, never an accounting of it, because any pair of changes can create a contradiction that neither carries alone. Bisection names which change carries most of the damage and it does not predict the total, so the three changes you re-apply have to be measured together before they go back in.

Bisecting four changes costs four extra runs, and the figure is worth taking apart rather than accepting. Each re-applied change is scored once, so four changes are four runs, and each run makes one call per case. Four runs over 400 cases is four times four hundred, which is roughly 1,600 calls. Eight-way concurrency sends those out in about two hundred rounds of eight, and at a few seconds a round that is the twenty minutes. The $6 is the same 1,600 calls at a small model's rate, which works out at a fraction of a cent each.

Set against a 12-point regression sitting in production, or against an afternoon of three engineers arguing about which change was responsible, the arithmetic is not close. The reason people skip it is that reverting feels like losing ground when three of the four changes were fine, and the answer is that you get them back within the hour with a number attached to each.

Proxies, drift and gaming#

Everything above produces a number on a set of cases, and the number is a proxy. It goes stale, because the distribution of real inputs moves — a product launch, a new integration, a support macro that changes what tickets look like — and an evaluation set assembled in January is describing January. Refresh it on a schedule, add cases from production incidents as they occur, and treat a set that has not changed in a year as evidence about a world that no longer exists.

The proxy is also gameable in the specific sense that anything optimised hard enough against a fixed metric starts to satisfy the metric rather than the goal. A prompt tuned across thirty iterations against an exact-match label set will acquire clauses that exploit quirks of that label set — a tie-break that happens to match how one annotator resolved ambiguous tickets — and those clauses do nothing for a user. The defence is a metric stack rather than a single number, and periodic human review to check that the score still means what it meant.

There is also a limit on what the discipline is for. A typo fix, a rewording that changes no requirement, a shortened sentence that says the same thing — running three seeds on 400 cases for these is theatre, and a team that demands it will find the demand quietly abandoned within a month. The gate belongs on changes that alter a requirement: constraints, formats, examples, task wording, model version. Everything else can ride on schema validity and a smoke test.

What the four rules buy, in the end, is not certainty about a prompt but the ability to be wrong in public and recover. A number attached to a fixed set makes a regression attributable, a revert cheap, and an argument short, and each of those is worth more than the number's absolute value ever is. The score is not the quality of the prompt; it is the thing that lets two engineers disagree about the prompt and find out who was right by Thursday.

None of this yet says anything about what a prompt costs to run, and that is the pressure most likely to arrive next. A 3,100-token prompt at 400,000 calls a day is a line item somebody in finance can see, and the request to halve it will come with the assumption that a shorter prompt is a free win — an assumption the apparatus you just built is the only thing capable of testing.