Designing a position experiment

Moving a rule to the end of a prompt and watching compliance rise is a claim about causation, and the evidence usually offered for it is two numbers. The numbers come from a compliance harness — a fixed set of stored cases replayed through the prompt offline, with a scorer that decides for each response whether the rule held — run once against the old prompt and once against the new one. The mechanism from chapter 1 makes an eighteen-point movement entirely plausible, and plausible is the problem: the harness runs a sampled generation over a set of cases, so it produces a different number every time it runs. If its own spread is four points, 71 and 89 are two measurements of two genuinely different things. If its spread is nine points, they might be two measurements of the same thing.

The experiment that tells those apart is built here on a redaction rule — any personal identifier appearing in a retrieved document must be redacted before it appears in a summary — sitting in the middle of a 12,000-token prompt, between the output-format section and the escalation policy. The vocabulary worth fixing first is the arm: one layout, run over the whole case set. Two arms is the version that cannot support the conclusion; three is the version that can.

Where the harness's spread comes from is worth knowing before deciding how to handle it, because a reader who does not know is likely to look for it in the wrong place. The number the harness reports is a fraction: the share of cases in which the rule held. Each case is decided by one sampled generation, and a handful of cases sit near the boundary where the rule may hold or fail depending on which of two nearly equal continuations gets drawn. Those are the cases that change verdict between runs, and each one that changes moves the reported fraction by one over the number of cases. On the frozen set below, that is a little under half a point per case. A standard deviation of a little over one and a half points is therefore what three or four boundary cases flipping looks like, which is a small and entirely ordinary thing to happen, and it is the whole reason a single pair of numbers cannot be read.

What has to be held fixed, and what cannot be#

An experiment that supports "position caused this" needs the two conditions to differ in position and nothing else. That sounds obvious and is violated routinely, in three specific ways.

The first is that moving a line and restating a line are different interventions. Chapter 1's remedy kept the full rule set at the top and added a short copy at the end, which changes both the position of the constraint and the number of times it appears. If the experiment compares "rule in the middle" against "rule in the middle and at the end", a positive result is consistent with position mattering, with repetition mattering, or with the sixty extra tokens of emphasis mattering. Three arms cost 50% more than two and separate all three: the original layout, the rule moved to the end, and the rule duplicated at the end.

What "separate all three" means is a set of subtractions, and the subtractions only exist because the third arm does. The moved arm minus the original isolates position, because those two layouts contain the same tokens in a different order. The duplicated arm minus the moved arm isolates what repetition adds, because those two differ by the presence of a second copy and by nothing else. The duplicated arm minus the original is the change that actually ships. A two-arm experiment measures only that last difference, which is why a two-arm result can be reported as a position effect while being mostly a repetition effect, or the reverse, with nothing in the numbers to say which.

The second is the case set. Both conditions must run over the same cases, in the same order, drawn once and frozen. Re-sampling production traffic for the second run makes the comparison a comparison of two weeks of traffic as much as of two prompts, and traffic composition moves — a marketing campaign that brings in a different customer segment will change the fraction of cases that contain a redactable identifier at all, which changes the denominator of the metric without anything about the prompt changing.

The third is the request itself. Everything that goes into the API call other than the line being moved has to be byte-identical between arms: model string, max_tokens, effort level, thinking configuration, tool definitions, and the order of every section that is not the one under test. This is where a subtle contamination enters, because on current Anthropic models you cannot pin the sampling. The decoding parameters — the knobs that control how the next token is drawn from the distribution the model produces — temperature, top_p and top_k, are rejected outright on Claude Opus 5, Claude Opus 4.7 and 4.8, Claude Sonnet 5 and Claude Fable 5; sending any of them returns a 400. On the older models that still accept temperature=0, setting it never guaranteed identical outputs either. The consequence is not that the experiment is impossible; it is that variance cannot be engineered away at the request level and has to be measured instead, by repeating each condition.

The instinct that temperature=0 buys a repeatable run is worth naming, because it is why variance goes unmeasured on the models that still accept the parameter. It is tempting for a good reason: zero temperature means always taking the highest-scoring next token, and always taking the highest-scoring anything sounds deterministic by construction. What it misses is that "highest-scoring" is a comparison, and a comparison between two candidates that are nearly tied is settled by a difference far smaller than anything a person would call meaningful. One such tie landing the other way early in a response changes every token after it, because each token is conditioned on the ones before. Greedy decoding therefore narrows the spread rather than removing it, and the design here does not depend on which of those two it does: repeat each condition, and report what the repeats did.

CONDITIONS = {
    "middle": build_prompt(rule_at="middle"),   # production layout
    "moved":  build_prompt(rule_at="end"),      # same tokens, different position
    "dupe":   build_prompt(rule_at="both"),     # +58 tokens at the end
}

def sweep(cases, runs: int = 5) -> list[dict]:
    """Same frozen cases, same request fields, one field varied: the layout."""
    records = []
    for name, system_prompt in CONDITIONS.items():
        for run_id in range(runs):
            for case in cases:                       # identical order every time
                r = client.messages.create(
                    model=MODEL,                     # pinned, not an alias in flux
                    max_tokens=1024,
                    output_config={"effort": "medium"},
                    system=system_prompt,
                    messages=case.messages,
                )
                records.append({
                    "condition": name,
                    "run": run_id,
                    "case_id": case.id,
                    "redacted": no_identifier_leaked(r.content[0].text, case.pii),
                })
    return records

The predicate — the function that reads one response and returns whether the rule held on it — deserves as much scrutiny as the layout. no_identifier_leaked here is an exact search for each identifier the case's documents contain, in each of the formats the identifier could plausibly be written in: the raw string, the string with separators stripped, and the last four digits presented as if they were the whole value. A predicate that only checks the raw string will score a response that reformatted the number as compliant, and a judge model would introduce its own error rate on top of the effect being measured. The measurement is only as trustworthy as its cheapest, most mechanical component.

Five runs per condition, and the number that was missing#

The frozen set is 300 summarisation cases, of which 212 contain at least one redactable identifier; compliance is the fraction of those 212 with no leak. Five runs of three conditions is 4,500 calls at roughly 12,000 input tokens each, so about 54 million input tokens — on the order of $160 at $3 per million before any caching, which is the entire cost of knowing whether the convention is real.

Each multiplication there is worth seeing, because the same factors decide the cost of every experiment of this shape. Three conditions times five runs is fifteen passes over the case set, and fifteen passes over three hundred cases is the four and a half thousand calls. Every call carries the whole prompt, so the input volume is that call count times twelve thousand tokens, which is the fifty-four million. Pricing that at three dollars per million gives the figure above. Note what is absent from the calculation: output tokens, which barely register because the responses are short summaries, and the conditional structure of the metric, which costs nothing because every case runs in every arm whether or not it contains an identifier to redact.

The original claim survives, and now it means something. The middle condition's best run is 73.6% and the moved condition's worst is 85.4%, so the two distributions do not overlap across five runs, and the sixteen-point mean gap is roughly ten standard deviations of the run-to-run noise. The duplicate condition adds a further 2.5 points over the moved condition — smaller than the position effect and larger than the spread, so repetition contributes something on top of position, which is exactly the decomposition the two-arm version could not produce.

All three of those readings are arithmetic on the table, and doing them is cheaper than trusting them. The mean gap is 87.5% minus 71.5%, which is the sixteen points. The run-to-run standard deviation in each of those two columns is 1.6 points, so dividing the gap by the spread gives ten: the difference between the layouts is ten times the size of the wobble that would have to account for it if position did nothing. The non-overlap reading needs no division at all — the best the middle layout managed in five tries was 73.6% and the worst the moved layout managed was 85.4%, so no single run of one could be mistaken for a run of the other. The duplicate column is where the arithmetic stops being comfortable: 90.0% minus 87.5% is 2.5 points against spreads of 1.1 and 1.6, which is outside the noise and not far outside it, and a result that size is one a sixth run could soften.

Pairing is sharper than comparing means for a reason worth stating plainly, because it is the same reason the case set had to be frozen. Cases differ enormously in difficulty — some put an identifier in the opening line of the first document, some bury a partially formatted one at the end of the last — and comparing two means folds all of that difficulty into the noise on both numbers. Comparing each case against itself under two layouts removes it, because whatever made a case hard is present in both arms and cancels. What survives the cancellation is only the cases the change actually moved, which is the quantity the experiment was asking about to begin with.

Because both arms ran the same 212 cases, there is a sharper test available than comparing means. Ignore the cases that both conditions redacted correctly and the ones both leaked, and count only the disagreements: in run 1, 39 cases went from leaking to redacted and 6 went the other way. Under the null hypothesis that position does nothing, those 45 flips should split evenly, and 39 out of 45 has a two-sided binomial p below 10⁻⁵, meaning a split that lopsided would turn up by chance less than once in a hundred thousand experiments.

That null hypothesis is doing more work than the sentence makes it look. It says that if the layout is irrelevant, a case disagreeing between the two arms is disagreeing for reasons of sampling alone, and a case of that kind is as likely to have moved one way as the other. Forty-five coin tosses would be expected to land near twenty-two or twenty-three improvements. Thirty-nine is not a near miss on twenty-two, and the very small p-value is a statement about how far from it the result sits rather than about how large the improvement is — a change that flipped four cases one way and one the other would be about as unlikely under the same null and worth a small fraction as much in production.

The six regressions are the part nobody looks at, and they are worth looking at: four of them were cases where the identifier appeared only in the final document, close to the moved rule, and the model redacted the identifier and then also redacted a product code that merely looked like one. That is a real defect introduced by the change, invisible in a sixteen-point headline.

What a failure to replicate looks like#

The same harness run on a different rule in the same prompt — a formatting requirement that tables must carry a units row — produced this instead: middle 74.6% with a standard deviation of 3.1 points, moved 79.2% with 4.4 points, and a paired flip count of 22 improvements against 17 regressions across the run. The means differ by 4.6 points and the distributions overlap heavily; the worst moved run scored below the best middle run. Nothing here says the effect is zero. It says the experiment cannot distinguish it from zero at five runs, and that detecting a 4.6-point difference against a 4.4-point spread would need something closer to twenty runs per condition, which is 18,000 calls for a result whose practical value is small.

Twenty is not a rule of thumb pulled from the air, and the reasoning behind it is the one to reuse. The spread of a single run is whatever the harness gives you; the spread of an average of several runs shrinks with the square root of how many are averaged. A 4.6-point gap only becomes legible once the averaged spread is down around a point, and getting a 4.4-point spread down to about a point means dividing it by roughly four and a half — which, because the shrinking goes as a square root, takes about twenty runs. Multiplying twenty runs by three conditions by three hundred cases is where the 18,000 calls come from. The same calculation run before the sweep rather than after it is how a team decides not to bother.

There is a second, more human failure mode that the design defends against. A single-run 71 → 89 is usually produced by running the new layout, seeing 81, adjusting the wording, seeing 84, adjusting again, seeing 89, and reporting the last one against a baseline measured once. Every intermediate run is a look at the evaluation set, and after four looks the reported number is a maximum over samples rather than an estimate of a mean. The defence is procedural: fix the layouts before the first run, run all conditions in the same sweep, and report every run.