Pre-registering a threshold

Everything the previous chapters built is machinery for producing evidence: the ablation, the churn floor, the per-case read, the instrumented rows, the knowledge of which eval can see a change, the curve that says whether volume is hurting. None of it produces a decision. Two engineers with opposite intuitions about where a block belongs can both read the same run and both find support, because a run of 500 cases can be summarised in a dozen legitimate ways.

The failure that follows is not dishonesty and it is worth being precise about that. A run of five hundred cases offers exactly that flexibility. You can report the aggregate, or the aggregate excluding the slice everyone knows is noisy, or the discordant split, or the per-slice table, or the subset of cases the judge was unanimous on. Every one of those is a legitimate view, and earlier chapters argued that you should look at several. Looking at several views is right; deciding which view counts after you have seen all of them is what turns a measurement into a rationalisation.

An opinion becomes a variant when it has a name and a file#

The first move is to stop calling them positions and start calling them variants. examples-first and evidence-first are two orderings of the same four blocks, written as two arm specifications differing in the block_order field, and once they are written down as files in evals/arms/ the argument is no longer about attention mechanisms — it is about which of two runnable configurations scores better on a set both sides agreed to.

Naming does a second thing that is easy to undervalue. A variant with a name can be referred to six months later by somebody who was not in the room. "We went with examples first" is a sentence about an outcome; examples-first@support-eval-v3 is a pointer to a configuration, a set, a hash and a number.

The threshold is the part that has to come first#

Pre-registration — writing down the hypothesis, the outcome measure and the decision rule before collecting data — arrived in medicine because trials that reported the outcome that happened to look good were producing a literature that did not replicate. The International Committee of Medical Journal Editors made prospective registration a condition of publication in 2005, and psychology followed a decade later with registered reports, where a journal accepts the study design before any data is collected. The problem being solved in both cases is that a data set of any size offers many defensible summaries, and a person who chooses among them after seeing the numbers will choose the one that agrees with what they already believed, without ever experiencing the choice as a choice.

The threshold that prevents this has four parts, and all four have to be written down while the answer is still unknown. There is the metric and the set it is computed on. There is the minimum effect that counts as a win, which cannot be smaller than the churn floor and is usually a multiple of it. There is the operational constraint the winner must respect, expressed as a number with units. And there is what happens on each outcome, including the outcome where the result lands between the two — because "inconclusive" needs a defined action, otherwise it silently becomes "ship it".

# evals/decisions/2025-08-19-fewshot-ordering.yaml
question: Do few-shot examples belong before or after retrieved passages?
set: support-eval-v3           # 500 cases, sha256 9f3c1ab7d2e04c85
churn_baseline: 4              # at 3 repeats, measured 2025-08-12
variants:
  - name: examples-first       # incumbent
    arm: evals/arms/order-a.yaml
  - name: evidence-first
    arm: evals/arms/order-b.yaml
threshold:
  metric: judge_helpfulness_mean
  win_requires: ">= +0.015 aggregate AND discordant split with McNemar p < 0.05"
  constraint: "p95 latency <= 5.0s AND cached-prefix tokens >= 3,000"
on_inconclusive: keep incumbent, record the numbers, do not re-open for 2 quarters
decided_by: [rota, kemi, dan]
agreed_at: 2025-08-19T10:20:00Z

The cached-prefix tokens line counts the unchanging head of the request, the part a provider can keep in a prompt cache and bill at a fraction of the usual rate. It is not decoration, and it is the reason this particular question has an answer that has nothing to do with attention.

What the run said, and what the cache said#

Both arms ran at three repeats with everything else pinned, on the same frozen set.

The quality column does not clear the threshold. Five thousandths of the score scale is a third of the agreed minimum, and the discordant split of 24 against 13 returns p = 0.10, which is the kind of number that looks like a trend and is not one. On the pre-agreed rule the answer is "inconclusive, keep the incumbent", and the interesting part of the run is in the last two rows.

Both halves of that verdict are checkable from the table without trusting anyone. The two judge means differ by five thousandths of the scale and the rule demanded fifteen thousandths, so the observed gain is a third of what was agreed; that is a division and it needs no statistician. The p-value is the McNemar statistic from the chapter on churn baselines applied to twenty-four improvements against thirteen regressions: the two counts differ by eleven, the continuity correction leaves ten, squaring gives a hundred, and dividing by the thirty-seven cases that moved puts the statistic under three. Its square root is a little over one and a half standard deviations away from an even split, and a departure that size turns up by chance about one time in ten. That is the number in the table, and "about one time in ten" is exactly the shape of result that reads as a trend to somebody who would like one.

Prompt caching matches on a prefix, which means the cached region runs from the start of the request up to the first byte that differs between requests. Anthropic's Messages API renders tools, then system, then messages, and a cache entry is only created if the prefix is at least 1,024 tokens on this model. In examples-first the system block and the few-shot block are byte-identical on every request, giving a stable prefix of 3,300 tokens that sits comfortably above the minimum and is read from cache at roughly a tenth of the base input price. In evidence-first the retrieved passages — which differ on every request by construction — sit second, so everything after them is uncacheable, and the only stable region left is the 900-token system block, which is below the minimum and therefore silently caches nothing at all. The API does not error on this. It returns cache_creation_input_tokens: 0 and charges full price, which is why the constraint was written as a number the harness could check rather than as an intention.

The word carrying that whole result is prefix, and its consequence is stronger than it first sounds. A cache entry is keyed on an exact run of tokens beginning at the very start of the request, so a lookup matches only up to the first token that differs from what was stored, and everything past that point is a miss — whether or not it is byte-identical to what was cached, because what the model computes for a token depends on every token in front of it, so a block that has shifted position cannot reuse anything. Order the blocks with the unchanging ones first and the matching run extends through all of them, ending where the retrieved passages start. Put the passages second and the matching run ends where they start, which is immediately after the system block, and the few-shot block sitting behind them is recomputed on every single request despite never changing a character.

The money follows directly. Moving 3,300 tokens per request from $3.00 per million to $0.30 per million saves $0.0089 a request, and at about 40,000 requests a day evidence-first costs roughly $356 a day more than the incumbent — about $10,700 a month — to buy an effect the eval cannot distinguish from zero.

That is three multiplications, and they are worth doing rather than accepting. Cached input is billed at roughly a tenth of the base rate, so each token that stops being cached costs the difference between the two prices, which is nine tenths of the base rate per million tokens. Multiply that per-token difference by the three thousand three hundred tokens that stop being cached and you have the per-request figure. Multiply the per-request figure by the forty thousand requests a day and you have the daily one. Multiply the daily one by a month and you have the monthly one. Nothing in that is subtle, and the reason to write it out is that this was the only column in the run that decided anything — the quality columns were inconclusive under the rule the team had already agreed, so the whole decision rests on a constraint somebody thought to write as a number.

Pricing 1.5 points#

The same machinery answers a question that feels quite different. A proposed change adds a cross-encoder reranker — a model that scores each retrieved passage jointly with the question rather than comparing pre-computed embeddings, which is more accurate and costs a forward pass per candidate — over 20 retrieved candidates, plus a second retrieval pass against a policy index. The team pre-agreed a threshold of +0.020 aggregate or a win concentrated in a named slice, subject to the existing p95 budget of 5.0 seconds.

Stated as a trade, the proposal buys a move from 0.83 to 0.845 for 4,000 more input tokens and about 600ms of p95. Read against the threshold, it is a clear no on two independent grounds: 1.5 points is short of the agreed 2.0, and 5.24 seconds breaches a latency budget that exists because the agent is called synchronously from a chat widget where a user is watching a spinner. The discordant split is real — p = 0.025 says the change genuinely helps more cases than it hurts — and a team without a pre-agreed rule would very likely ship it on the strength of that p-value, then spend the next quarter explaining a latency regression.

Three of the figures in that reading are subtractions from the table and one is a multiplication, and a reviewer who trusts nobody can reproduce all four. The judge mean rises by fifteen thousandths of the scale, which is one and a half points, against an agreed minimum of twenty thousandths, which is two — so the proposal delivers three quarters of the bar it had to clear. Mean input tokens rise by four thousand. The p95 rises by a little over six tenths of a second, past a budget that was set in whole seconds. And the marginal spend follows from the token column alone: four thousand extra input tokens on forty thousand requests a day is a hundred and sixty million extra input tokens a day, and at the base rate per million that is the figure in the last row.

The per-slice read makes the outcome more useful than a flat rejection, and the count it turns on is a net rather than a raw one: thirty-one cases improved and fifteen regressed, so sixteen cases' worth of improvement is what the aggregate column is made of. Eleven of the sixteen net improvements land in warranty/edge-case, which is the slice where the right policy paragraph is genuinely hard to retrieve and is 8% of production traffic. Reranking only on that route is a third variant, and it was evaluated under the same threshold: aggregate 0.841, p95 4.68 seconds, marginal spend $38 a day. That clears the constraint and lands short of the quality bar, so it ships as an experiment with an agreed review date rather than as a permanent change — which is a decision the record can state plainly, including the fact that it was a judgement call at the boundary.

A business rule is not a significance test#

A pre-agreed threshold is not a significance test and does not replace one. It is a business rule about how large an effect has to be to be worth its cost, and the statistics tell you separately whether an effect that size is distinguishable from noise. The reranker cleared the statistical bar and failed the business one; a change can equally clear the business bar with a number the harness cannot distinguish from churn, which is the case where the honest response is more repeats rather than a decision.

Nor does pre-registration forbid looking. The per-slice analysis that produced the third reranker variant was not in the original record, and it should not have been — nobody knew to ask about warranty/edge-case until the discordant cases were grouped. What pre-registration requires is that findings discovered after the fact are labelled as what they are: hypotheses for the next pre-registered run, not results of this one. The third variant earned its place by being re-run against its own threshold, not by being spotted in the data.

The rule also cannot rescue a metric that measures the wrong thing. If the judge rewards thorough answers and the business needs short ones, a threshold on judge mean will faithfully select for the wrong system, agreed in advance and recorded impeccably. And a threshold has a shelf life for the same reason the churn floor does: a new model snapshot changes what a point costs and what a point is worth, so a decision recorded against claude-sonnet-4-5-20250929 is evidence about that snapshot.

Two named variants, a metric, a minimum effect, a constraint in numbers, and a defined action on every outcome — including the one where nothing is distinguishable — is the smallest apparatus that converts a run into a decision. It costs about fifteen minutes before the run and it is the only fifteen minutes in which the rule can be agreed honestly. Recording the result next to the arm files makes revisiting a decision a deliberate act with a starting point, which is the difference between revising a decision and re-having an argument.

All of this answers questions of the form "is this specific change worth it", one block at a time. The question a director asks is larger and shaped differently: not whether to add the reranker or move the few-shot block, but what fraction of the 14,000 tokens on every request is doing any work at all. That question has a defensible answer, an obvious wrong answer, and a trap in between.