Rollback thresholds and ship gates

The usual framing of an unmeasured deploy is process discipline, which makes it sound like paperwork and invites the reply that the team moves fast. The concrete loss is a decision procedure. Once a change has shipped without a before-number, a later measurement of 0.641 is a number with nothing to be compared to: no threshold can be written, no alert can fire, and reverting becomes an argument rather than a calculation. Without the second of the three terms, the rule has no truth value — "0.641 is below the threshold" is not a sentence that can be true or false when no threshold exists, and no amount of care taken over the measurement supplies one.

Three separate capabilities disappear at once, and they are worth naming separately because they fail at different moments. Proof goes first: you cannot demonstrate that quality dropped, only that it is now 0.641. Attribution goes next: even if everyone agrees quality dropped, six other things shipped in between, and without a per-change before-number none of them can be isolated the way the previous chapter isolated an embedding model. Action goes last and hurts most: reverting is itself a change with a cost — documents re-chunked, a full re-index, downstream code that assumed the new shape — and nobody signs off on that cost against a suspicion.

Three cheap dimensions and one expensive one#

Three of those four dimensions can be measured on any deploy with no labelling at all, because latency, token count and index size need no judgments. Only the first row requires a golden set, and it is the row a quality argument turns on. A team that gates on the three cheap dimensions and skips the expensive one still has a rollback threshold for most of the ways a chunking change goes wrong, which makes the cheap gate the one to build first rather than the one to build eventually.

The token row is where the money is, and it is the row that goes unnoticed longest because no dashboard carries it by default. At 4,300 prompt tokens instead of 1,900 and roughly 41,000 assistant answers a month, that change added about 98 million prompt tokens monthly; at $3 per million that is a shade under $300 a month, which nobody would have blocked on had it been visible and which nobody could see. It is worth measuring anyway because it compounds with every later change — a reranker that passes twelve chunks instead of eight, a system prompt that grows — and the only way to know which change owns which part of the bill is the same per-change before-number the quality argument needs.

Both steps of that costing are worth doing rather than quoting, because the shape of the result is what makes the row easy to ignore. The extra tokens are 4,300 minus 1,900, which is twenty-four hundred per answer, and twenty-four hundred across roughly 41,000 answers a month is a little over 98 million tokens. At $3 a million that is under $300. Nobody blocks a deploy over three hundred dollars, and that is precisely why the row goes unwatched — it is individually too small to argue about and it is a ratio rather than an amount. The ratio is 4,300 over 1,900, which is more than double, and a dimension that doubles is a dimension that will double again. Two more changes of the same size take the same volume of answers past a bill somebody does convene a meeting about, and by then three changes share the blame and none of them has a before-number.

The objection is half right#

The engineer defending an unmeasured change makes an argument that deserves a real answer: the corpus in June is not the corpus from March, so a March baseline is not a valid comparator, and holding up a stale number as if it were one would be misleading. That is correct, and the previous two chapters are largely about why it is correct. A number produced against a different corpus snapshot, a different query set or a different qrels version measures a different thing, and 0.610 against 0.641 across six months of ingestion is not a comparison.

Where the argument goes wrong is the conclusion. The staleness of the absolute number is an argument for replaying the old configuration, not for measuring nothing, and the replay is cheap precisely because the archived manifest from the freezing chapter contains everything needed to reconstruct it. Both configurations get scored in the same session, against the same corpus snapshot, the same query set and the same judgments, and the comparison is a paired per-query delta rather than two historical numbers held next to each other.

The second row is the cheapest thing worth accepting, and it is what to ask for. If the golden set has no current judgments, the third row is the fallback and it is smaller than it sounds: only the queries where the two configurations return different documents in the top ten need judging, which on a chunking change is typically 30 to 50 of 140 queries and a single afternoon. The worked case below lands just past the top of that band, at fifty-two, and the reason is worth having: a chunking change reshapes every document in the index, so it perturbs more queries than a change to a scoring parameter would. The band is a property of the kind of change rather than of the query set, and a re-ranking weight adjustment would sit at the bottom of it. Where the surface has real traffic, team-draft interleaving is worth knowing about — it merges the two rankings into one list, attributes each click to whichever system contributed the document, and detects a preference with roughly an order of magnitude less traffic than a conventional A/B test, without any labelling.

What none of these permits is the fourth row as the price of entry. Demanding a full labelled evaluation before any retrieval change moves is how teams end up shipping unmeasured, because the demand is expensive enough that people route around it. The gate has to cost less than the change it is gating.

The diff-set technique is worth spelling out, because it is the one that rescues a team with no current judgments and it is not widely practised. Run both configurations over the query set, take the symmetric difference of their top-10 document sets per query, and judge only those documents. Queries where both configurations returned the same ten documents in the same order contribute an identical score to both runs whatever the judgments say, so they can be skipped entirely, and queries where the sets overlap heavily need only a handful of new judgments. On one chunking change the two configurations agreed completely on 88 of 140 queries and the remaining 52 produced 611 unjudged document-query pairs, which one labeller cleared in about three hours. The resulting numbers are not a baseline — they are valid for this comparison and nothing else, because the judgments are complete only with respect to these two runs — and that is enough to answer the question in front of you. Recording them as qrels-v5-diffonly rather than promoting them into the main file keeps the distinction honest.

The two counts in that example also say how different the configurations really were, which is information the comparison itself does not surface. Eighty-eight queries agreed completely and fifty-two did not, which is 140 in total. Across those fifty-two, 611 pairs needed judging, a bit under twelve per query. A symmetric difference of twelve between two lists of ten means each configuration held six documents the other did not, so on the queries where they disagreed at all, they agreed on four of their top ten and differed on the rest. That is a large disagreement, and it is the number that tells you the comparison is worth running: two configurations agreeing on nine of ten would produce a delta too small to separate from noise however carefully it was judged.

Why the skipping is sound rather than merely convenient takes one step. A query where both runs returned the same ten documents in the same order produces the same gain at the same discount at every position for both, whatever grades a labeller would assign. The two per-query scores are therefore identical by construction, and identical numbers contribute nothing to a paired delta. Judging those documents would cost a labeller's afternoon and move the answer by exactly zero.

Gating each dimension separately#

There is no scalar that combines quality, latency and cost into a single verdict, and attempts to build one produce a weighted sum whose weights are argued about forever. Gating on each dimension separately avoids the argument: a change that improves nDCG@10 by 0.03 and doubles p95 latency fails, and the conversation that follows is about whether the latency budget is right rather than about whether 0.03 of nDCG is worth 170 ms. Where the trade genuinely has to be made, it gets made by a person with the numbers in front of them, once, and the outcome becomes the new threshold.

# eval/ship-gate.yaml — checked in CI against the replayed baseline
baseline: baselines/bm25-hybrid-2024-09-14   # replayed on today's snapshot
noise_band: 0.012                            # from the 10-run spread
gates:
  - metric: ndcg@10
    rule: delta >= -0.012                    # within noise is acceptable
    on_fail: block
  - metric: recall@50
    rule: delta >= -0.010                    # candidate ceiling must not drop
    on_fail: block
  - metric: p95_latency_ms
    rule: ratio <= 1.25
    on_fail: block
  - metric: prompt_tokens_per_answer
    rule: ratio <= 1.5
    on_fail: warn
  - metric: ndcg@10:stratum=error
    rule: delta >= -0.030                    # per-stratum floor, thin sample
    on_fail: warn
rollback_trigger:
  window: 14d
  rule: ndcg@10 falls more than 0.02 below the value recorded at ship time

$ python -m eval.gate --config eval/ship-gate.yaml --run runs/chunk-1500-200
ndcg@10                   0.6412 -> 0.6288   delta -0.0124   BLOCK
recall@50                 0.9310 -> 0.9295   delta -0.0015   ok
p95_latency_ms             141.0 ->  308.0   ratio  2.18     BLOCK
prompt_tokens_per_answer    1907 ->   4302   ratio  2.26     WARN
ndcg@10:stratum=error     0.5980 -> 0.5410   delta -0.0570   WARN
2 blocking failures

The noise band in that configuration is the run-to-run spread from the previous chapter, and it is what makes the first gate honest rather than paranoid: a change is blocked for moving the metric further than the system moves on its own, not for moving it at all.

Reading the output against the rules shows which of the five gates is actually deciding, and it is not the one people would quote. The quality gate fails by four ten-thousandths — a delta of 0.0124 against a band of 0.012 — which is a block earned by the narrowest possible margin and a result that would flip on a different pair of runs. The latency gate fails by a mile: a ratio of 2.18 against a ceiling of 1.25, which no re-run is going to reverse. The recall gate passes cleanly, so the candidate ceiling is intact and this is a pure ordering-and-cost change. Two of the five say block, which is the count printed at the bottom, and the two warnings are recorded without stopping anything. What that layout buys is a decision that does not depend on the expensive measurement: strike the quality row entirely, as a team with no golden set would have to, and the deploy is still blocked, on a number that took no judgments to produce.

Where the gate stops being the right instrument#

A passing gate is not evidence that the change helps users. It says the offline proxy did not detect harm on 140 judged queries drawn from last quarter's traffic, which leaves substantial room for a change that damages a query type the strata under-sample, or that degrades the experience in a way retrieval metrics do not encode — stale documents ranked highly, or chunks that are individually relevant and collectively redundant. Monitoring after the deploy remains necessary, and the rollback_trigger block exists because the gate's judgment at ship time is provisional.

The thresholds are also not permanent. The noise band came from a ten-run spread on one stack, and it changes when the index topology changes, when a hosted reranker is introduced, or when the golden set grows. A gate whose numbers were set eighteen months ago and never revisited is blocking on a band that no longer describes the system, which produces the worst failure mode available: a check that people learn to override.

And the rule has honest exceptions. A pure revert to a previously measured configuration does not need a new gate run, because the before-number already exists on the other side. A change to a surface with no traffic can ship and be watched. What does not qualify as an exception is a change that reshapes every document in the index and whose effects are diffuse enough that nobody will notice them for a quarter.

A before-number is the cheapest artefact in this topic and the one whose absence is most expensive, because it is the only thing that converts a disagreement into a comparison. Three of the four dimensions it covers cost nothing but a script, the fourth costs a replay of forty minutes against an archive that already exists, and the fallback when no current judgments exist is an afternoon of judging the queries where two configurations actually disagree.

Everything in this topic exists to be run before the interesting work begins. A lexical retriever, evaluated against a stratified golden set with complete judgments, scored on recall at the candidate depth and nDCG at the consumed depth, frozen with its corpus snapshot and qrels and cutoff, and archived with its per-query results, is the number every subsequent claim about hybrid retrieval, query expansion, reranking or a new embedding model will be measured against. Building it takes a few days, and it is the difference between a retrieval roadmap that accumulates and one that produces improvements that do not add up.