Per-slice re-qualification

Re-qualification is the measurement that decides whether a replacement model can take over a feature from the version running in production now. A deprecation forces it on you, and the artefact it produces is nearly always a single number: the successor's score on your eval suite set against the incumbent's.

This chapter is about the cut that has to happen before that number means anything. A slice is a subset of the eval set defined by an axis the system actually varies along — a language, a document layout, a tool schema, a class of refusal — scored on its own. Slicing brings its own failure, which is reading noise as a finding, so the margin of error on each slice matters as much as the delta, and the pass criteria have to be fixed before anybody is looking at the numbers.

Three points of aggregate can hide a nineteen-point collapse#

An aggregate score is a weighted mean, and a weighted mean is a device for making a small number of large slices decide the result. Cut the same run by the axes the system actually varies along and the picture stops being a single number:

The 1,500 English items carry 63% of the weight and gained six to eight points each, which is enough to pay for a 19-point collapse on a slice a quarter their size and still show a gain. This is not a pathology of the successor model; it is arithmetic that applies to every weighted mean ever computed, and it is why "the score went up" is a statement about the mixture as much as about the model.

Working the arithmetic once, in words, turns it into something you can predict rather than something you check afterwards. An aggregate gives every slice a share of the total equal to its size — the 980 English invoices are 980 of the 2,400 items — and then adds up each slice's score multiplied by its share. The change in the aggregate is that same sum performed on the deltas: every slice contributes its own movement, scaled down by how much of the set it is. A slice that is a large part of the set passes nearly all of its movement through to the total, and a slice that is a small part of it passes nearly none.

That is the whole of why a small gain on a large slice outweighs a large loss on a small one. The English rows carry nearly two-thirds of the weight and moved six to eight points each, so most of six to eight points arrives at the total. The German row moved nineteen points, which is much further, but it is a tenth of the set, so only a tenth of nineteen points arrives. The larger contribution wins, the aggregate rises three points, and no step in the arithmetic has gone wrong.

What that leaves you holding is not a number about the model. It is a number about the model as seen through a mixture, and it is a property of both. Change the proportions of the eval set without touching a model or a label and the aggregate moves on its own; a suite with twice as many German invoices, scored against the same two versions, would have reported a fall.

The mixture in your eval set is also not the mixture in production, and rarely by accident: eval sets are built from whatever was convenient to label, then grow by accretion when someone adds cases after an incident. German invoices being 10% of the eval and 4% of production traffic sounds like the eval over-weights them, until you notice that the 4% is concentrated in three enterprise accounts and one of them is 31% German. An aggregate weighted by eval-set composition answers a question nobody asked.

None of that says why there was a collapse to hide in the first place, and that mechanism is what makes per-slice work necessary rather than merely careful. A successor is not the incumbent improved. It is a different set of trained numbers, produced by a different training run over different data, and the relation between the two is replacement rather than edit. Nothing in that process guarantees that a change helping one kind of input helps every kind, because no step of it was aimed at your kinds of input in the first place.

The expectation to carry into a re-qualification is therefore a redistribution rather than an increase: most things improve, some things get worse, and which things land in which group is not knowable from outside the training run. That expectation is exactly what makes an aggregate the wrong instrument, because absorbing a redistribution into one number is what a mean does. A single figure over a mixture cannot separate "everything rose a little" from "most things rose and one thing fell away", and those two results call for opposite decisions.

Quality by language is only one axis. The ones that break independently in practice, and therefore need their own row:

• each language or locale you support
• each document or content type, and each layout family within it
• each tool call — the model answering with a structured request to run one of your functions, which has to match the schema you declared for it — scored on schema validity separately from semantic correctness
• refusal and safety-decline cases, which can move sharply on a generation bump
• latency percentiles, not the mean
• cost per request

On the same run, the tool-call slice reported schema validity falling from 99.2% to 94.8%. That is a 4.4-point move on a metric where the interesting region is the last 1%, and it means one in twenty extractions now returns JSON your parser rejects. Latency told a similar story from a different direction: p50, the median request, went from 1.2s to 1.3s, which is nothing, while p99 — the latency that only the slowest one request in a hundred exceeds — went from 4.1s to 8.2s, which is past the 6-second client timeout and turns roughly 1.5% of requests into errors the eval score never sees, because the eval harness has no timeout.

A slice too small to move is a slice you cannot judge#

Slicing introduces a failure mode of its own, which is reading noise as signal. With 210 French items scoring around 0.85, the standard error is about 2.5 points and the 95% interval is roughly ±5 points; the +0.02 in that row is indistinguishable from nothing. The handwritten-annotation slice has 100 items and an interval of about ±9 points, which means it cannot detect any regression smaller than nine points — a slice that size is a smoke alarm, not a measurement.

Those intervals are worth understanding rather than trusting, because the shape of the formula is where the practical consequence hides. A slice score is a proportion: the share of that slice's items the grader passed. Each item is a single pass-or-fail outcome, so the score is a column of ones and zeros added up and divided by how many there are. A score built that way has a spread even when nothing about the model has changed, because another sample of French invoices of the same size, drawn from the same population, would have passed a slightly different count.

Two properties of that spread appear directly in the function below. It is widest when the score sits in the middle, since a slice where about half the items pass has the most room to land differently, and it narrows towards either end, since a slice where almost everything passes has almost nothing left to vary — that is the p * (1 - p). And it falls as the slice grows, but with the square root of the count rather than with the count itself, which is the sqrt and the n beneath it. The z is only how sure you have chosen to be, with 1.96 the conventional constant for a 95% interval.

The square root is the part with a consequence you can act on. Because the spread falls with the square root, halving a slice's uncertainty costs four times the labelled items rather than twice. That is why adding a few more German invoices does not rescue a slice you could not judge: moving an interval from unusable to usable is a labelling project with a size, and knowing that size in advance is the difference between planning it and discovering it.

The practical rule is to compute the interval per slice before reading any of the deltas, and to state the detection floor next to each row:

from math import sqrt

def moe(p, n, z=1.96):
    """95% margin of error on a proportion-style score."""
    return z * sqrt(p * (1 - p) / n)

for s in slices:
    m = moe(s.old, s.n)
    verdict = (
        "regression" if s.new < s.old - m else
        "improvement" if s.new > s.old + m else
        f"inconclusive (needs ±{m:.02f})"
    )
    print(f"{s.name:32} n={s.n:>4}  {s.old:.2f} → {s.new:.2f}  {verdict}")

Three outcomes rather than two is the point. A slice that comes back inconclusive has not passed; it has failed to be measured, and the responses to that are to add labelled items, to merge it into a neighbouring slice you are willing to judge jointly, or to accept explicitly that you are shipping without evidence on that axis. Any of those is defensible. Reading +0.02 as a small win is not.

That reading is a structural temptation rather than a careless one, and the table gives no help against it. A measured +0.02 and an unmeasurable +0.02 are the same characters in the same column: both small, both positive, both scanning as mild good news to somebody looking for red. What separates them is a quantity that appears beside neither of them unless somebody decided to print it, which is the argument for the verdict string in the code above rather than a footnote about statistical power somewhere else.

Computing that interval before the deltas, rather than after, is not fussiness either. An interval worked out once the deltas are known is chosen by somebody who can already see which side of it each row falls on, and the pressure at that moment runs one way — towards whichever interval lets the migration proceed. Computed first it costs nothing and commits nobody, because it depends only on each slice's size and the incumbent's score, and both of those are known before the successor is run at all.

A vendor's benchmark table is evidence about their eval#

The argument for skipping re-qualification usually rests on the provider's own comparison table, and that table is real evidence — of something else. Published benchmarks are aggregate scores on public academic sets, chosen to be comparable across models and vendors, which is exactly the property that makes them share nothing with your document distribution, your label taxonomy, your languages, or your tool schemas. "Ahead on every listed eval" is a claim about performance on those sets. It contains no information about German invoices at all, and it cannot, because nobody at the provider has seen one.

The mitigation usually proposed alongside it — ship the successor to 5% of traffic and watch the error rate — fails for a second, independent reason. A canary, meaning a small slice of live traffic served by the new version while everything else stays on the old one and watched for trouble before the rest follows, read through an aggregate error rate is the same weighted mean as before, now computed over live traffic and with less statistical power. A regression confined to one language at 4% of volume moves a global error rate by fractions of a point, sitting comfortably inside the day-to-day variance the on-call already ignores. Canaries are good at catching things that break loudly for everyone; the regressions that come with a model swap are usually the other kind.

That failure is worth naming rather than filing under bad luck, because a canary feels like the stronger check — real traffic, real users, real money — and an offline suite feels like the cautious substitute for it. The property that decides the matter is not realism, though, it is resolution. Any monitor can only report a change larger than its own ordinary movement, so the smallest regression a canary can find is fixed by how much its error rate wanders on a quiet week rather than by how much traffic passes through it. Below that floor the reading is not a late signal or a weak one. There is nothing there to see, and leaving the canary out for another fortnight does not put anything there.

None of which makes the vendor's table useless. It is a good prior on whether the successor is worth qualifying at all, and a reasonable input to choosing between two candidate models before you spend a week on either. It is an input to the decision to run the eval, not a substitute for its result.

The same price per token, thirty percent more money#

Quality is not the only axis that a swap moves silently. Two weeks after one such cutover, per-feature cost tracking showed extraction running 30% more expensive on the same request volume, against a successor whose published price per token is identical to its predecessor's.

The successor is more verbose. Median output on the extraction task went from 240 tokens to 385, on a request that sends about 1,200 input tokens:

That is $826 a day, about $25,000 a month, from a model swap whose price sheet did not change. Output tokens are priced at five times input on this family, so a change in verbosity moves the bill far more than the same proportional change in prompt size would. Reasoning-style models make this sharper still, since thinking tokens — the working-out a model generates for itself before its visible answer — are billed as output, and a successor that thinks by default on a task where its predecessor did not can multiply the output side without changing a single visible character of the answer.

Each link in that chain is worth making explicit, because it is what lets you predict the next case instead of re-measuring it. A provider charges per token rather than per request, so the price of a call is a rate multiplied by a length and not a fixed sum. The length of an answer to a given prompt is a behaviour the model acquired in training rather than a parameter on your side — instructions can push it, but the default belongs to the model, and a successor arrives with its own. Those two facts together are what allow a version change to move your bill with no action taken on your side and nothing announced on theirs. The five-to-one ratio between output and input pricing then settles how far it moves: the length the model decides sits on the expensive side of the bill, and the length you decide sits on the cheap one.

The mechanism is not exotic and the fix usually is not either — an explicit instruction about response shape, a structured output format, or a lower effort setting where the family exposes one. What matters for re-qualification is that cost per request is a slice like any other, measured on the eval run, before the migration rather than a fortnight after it. You already have per-feature and per-tenant cost attribution; the eval harness needs to emit usage per case so the same attribution can be computed on candidate models offline.

The suite scores a prompt and a model together#

There is a fairness question buried in any run that produces a result like the German one, and it cuts both ways. Your prompt was written against the incumbent, and much of it exists to compensate for that model's specific habits — the emphatic instruction added when it kept omitting a field, the format example added when it kept wrapping JSON in prose, the three sentences of hedging added after a bad week in March. Scoring a successor against that prompt measures the successor's behaviour under instructions tuned for something else.

The honest version of the comparison therefore has two runs, not one: the successor on the existing prompt, which is what shipping tomorrow would actually do, and the successor on a prompt given a fair attempt at re-tuning. When those two differ a lot, the gap is the size of the prompt-engineering task the migration implies, and it belongs in the plan rather than being discovered afterwards. Re-tuning frequently means deleting rather than adding, because compensations written for an older model tend to over-apply on a newer one that follows instructions more literally.

The trap in the other direction is tuning the prompt until the gate passes and calling that qualification. A prompt optimised against 2,400 eval items on a specific model is fitted to those items, and the score it produces has stopped being an estimate of production behaviour. If prompt work happens during re-qualification — and it usually should — it needs a held-out portion of the suite that the tuning never saw, or the gate is measuring your persistence rather than the model.

Refusal behaviour deserves its own note for the same reason. Safety classifiers move independently of the weights and independently of your prompt, and a generation bump can change decline rates on content that is entirely benign but adjacent to a restricted domain. If any part of your corpus touches security, health, or biology topics, a refusal slice with real examples from your own traffic is the only way to see it; the aggregate score treats a declined request as a low-scoring one and buries it.

Where the gate lives#

The output of all this is a set of per-slice criteria that a run either meets or does not, written down before the run:

# eval/gates/extraction.yaml
aggregate:
  min_score: 0.86            # necessary, never sufficient
per_slice:
  default:
    max_regression: 0.02     # beyond the slice's own margin of error
  de_invoices:
    max_regression: 0.00     # enterprise account depends on this slice
  tool_calls:
    min_schema_validity: 0.990
latency:
  p99_ms_max: 5000           # client timeout is 6000
cost:
  max_increase_per_request: 0.10
refusals:
  max_rate_increase: 0.005

One word in the sentence introducing that file does most of the work, and it is "before". A threshold written after the result is known is not a threshold, because whoever writes it can already see which side of it the answer falls on, and at that moment there is no counterweight: nobody argues for a stricter bar on a migration that is already late, and the honest reason for moving a number is indistinguishable from the convenient one when both are available. Written first, the same figure is a commitment made without knowing which way it will cut, and that is the only condition under which it can come back no. What the file buys is not machine-readability. It is that the criteria are older than the numbers they judge, so that nobody negotiates them after seeing the run.

An eval set ages out of resembling production#

An eval suite approves a model against the distribution it encodes, and that distribution ages. New customers arrive with layouts nobody has labelled, and a suite built eighteen months ago scores a model on a corpus that is no longer your corpus. Golden datasets — the curated cases whose correct answers a human has fixed in advance, which the suite scores everything against — need maintenance on a cadence, and Regression Testing for LLMs: Golden Datasets Explained is a reasonable starting point for what that maintenance involves; the failure mode to watch for is a suite that keeps passing because it stopped resembling production.

A suite runs at a moment in time#

The other limit is that a suite is not monitoring. It scores a fixed set of cases at one moment and says nothing about the request arriving now. Per-slice production metrics — the same cuts, computed continuously on live traffic — are a separate build, and a migration is exactly when you find out whether you have them, because a cutover without per-slice production visibility is a cutover you cannot evaluate after the fact.

An aggregate over a mixture nobody chose usually rises when the largest slices rise, so a team that approves on the aggregate has built a gate that approves. Per-slice results, with their margins of error stated and against thresholds written down in advance, produce a verdict that can come back no.

What such a verdict does not do is dispose of the deadline. The suite can say the successor is unshippable on a slice that matters; it has nothing to say about the fact that the incumbent is being switched off in four months regardless. That gap between "the replacement fails" and "the incumbent is going away" is the situation a migration plan has to be built to survive, and it is where the sequencing decisions get made.