When the golden set and production disagree, production decides

The offline suite, the golden set this team scores every candidate against, is 1,240 examples, hand-curated over about a year, scored by a rubric that four people maintain. It has picked the winner on every model swap since the project started, and the two times someone overrode it the override was wrong. On the current change it scores the incumbent configuration at 74.2 and the challenger at 66.1, a gap of 8.1 points that is stable across three re-runs.

The online test on the same change ran fourteen days, user-randomised, exposure logged at assistant open, 54,300 exposed users per arm. The challenger wins task completion by 5.0 points, interval [2.8, 7.2]. Spend is up 4%, well inside its 15% limit. p95 latency is up 60 ms against a 250 ms limit. The safety flag rate is 0.19% against 0.20%. No guardrail is breached.

Half the team wants to trust the suite and investigate the online result as an anomaly. The head of product wants a decision by Friday.

Why the year of track record does not settle it#

The suite's record is evidence that it predicted well on the changes it has seen. Every one of those was a model swap or a decoding-parameter change, and this one is a retrieval and prompt restructure that alters which passages reach the model. A proxy validated on one class of change carries no guarantee onto another class, and the accumulated confidence in the suite is confidence in a sampling of changes rather than in the construct.

The deeper point is that these are not two measurements of the same quantity with different noise levels. The offline suite estimates rubric score on 1,240 curated examples. The online test estimates the thing the rubric was built to approximate — whether real users finish what they came to do — on the real traffic distribution, with the real users, over a real window. Treating the online result as a noisy reading of the offline quantity inverts the relationship, and doing it repeatedly is how a proxy quietly becomes the goal: the suite stops being a cheap early signal about outcomes and starts being the outcome, at which point improving it and improving the product come apart.

So the decision is the challenger ships, and the justification is that the online test measured the outcome directly on the population the change will serve, and every declared limit held.

That justification is only available if the online result is sound, which is a checklist rather than an act of faith. The randomisation unit is the user and the product is experienced per user. The exposure rule fired on assistant open, evaluated identically in both arms and before any branch. The horizon covered two whole weeks and the lift by days-since-first-exposure was +5.9 in the first three days and +4.7 in days 8–14, so the win is not novelty. The analysis ran once at the horizon. And the traffic split matches what was configured:

from scipy.stats import chisquare

control, treatment = 54_612, 54_301     # units entering the analysis
expected = [(control + treatment) / 2] * 2
stat, p = chisquare([control, treatment], f_exp=expected)
print(f"SRM check: chi2={stat:.2f}, p={p:.3f}")
# SRM check: chi2=0.90, p=0.343   → no sample ratio mismatch

A sample ratio mismatch is the cheapest invalidation there is and the one most often skipped. A 50/50 split that arrives as 51.5/48.5 on a hundred thousand units has a p-value around 1e-13, and it means units were dropped or misassigned by something correlated with the arm, which usually means the treatment is crashing on a subset and those users are silently leaving the analysis. Any SRM makes the effect estimate uninterpretable regardless of how clean it looks.

Mining the disagreement instead of explaining it away#

The disagreement is data about the suite, and the way to read it is to line up the offline set's composition against production's.

The two slices where the incumbent wins offline by double digits are 4% and 9% of real traffic and 31% and 22% of the golden set. The two slices where the challenger wins big online are 43% of traffic between them and 11% of the set. The suite is not wrong about anything; it is weighted toward the cases the four maintainers found interesting to curate, which are the hard, well-specified, quotable ones, and away from the messy underspecified queries that dominate the actual product. The 8.1-point offline gap is almost entirely the policy-citation and long-thread rows.

That gives three concrete actions rather than a vague resolve to improve coverage. Reweighting the existing suite to production's slice mix moves the offline gap from −8.1 to −1.4, which is not agreement but is no longer a contradiction, and it costs one query. Sampling new cases from production traces in the two under-represented slices — 150 multi-intent and 200 vague-query examples, drawn from the experiment window, deduplicated and labelled — costs about a week of annotation and permanently changes what the suite can see. And a standing check that the suite's slice distribution is compared against production's each quarter catches the drift before it produces another 8-point surprise, since traffic mix moves on its own as the product acquires new users.

The disagreement itself gets written down, because the value of a register like this is entirely in it being consulted the next time.

# eval/disagreements/2026-03-retrieval-restructure.yaml
change: retrieval-restructure-v2
offline:
  suite: golden-v7 (1240 examples)
  result: incumbent +8.1 rubric points
online:
  experiment: support-retrieval-v2
  result: challenger +5.0 pp task completion [2.8, 7.2]
  guardrails: all within limits
  validity: SRM p=0.343; exposure at assistant_open; 14d; novelty checked
decision: shipped challenger
root_cause: >
  Golden set over-weights policy-citation (31% vs 4% of traffic) and long
  technical threads (22% vs 9%). Under-weights multi-intent (3% vs 19%) and
  vague single-line queries (8% vs 24%), which is where the challenger wins.
actions:
  - reweighted golden-v7 to production slice mix (gap −8.1 → −1.4)
  - added 350 harvested cases in the two under-represented slices → golden-v8
  - quarterly slice-distribution check added to eval CI

What would justify overriding the online result#

Four situations, and they are narrow.

The online measurement is invalid. An SRM, an exposure rule that turned out to depend on treatment output, a bug in the metric SQL, arm leakage from a shared cache, or an unrelated launch that landed in the same window on one arm's population. These are checked before the decision, not invoked afterwards when the result is unwelcome.

The primary metric is gameable and the challenger gamed it. Here resolved can be set by the assistant itself when it believes it has answered, so a configuration that is more willing to declare success moves the metric without helping anyone. The check is an independent signal that a gaming strategy would not move in the same direction: reopen rate within 7 days, customer satisfaction scores on resolved conversations, or repeat-contact rate. On this test, reopen rate fell from 8.1% to 7.4% and satisfaction was flat, which is inconsistent with premature closing. If reopens had risen, the win would be an artifact and the offline suite would have been right for a reason nobody had articulated.

A rare, severe failure mode that online traffic cannot power. The suite contains 60 adversarial cases about medical and legal advice boundaries, and the challenger fails 4 of them. Those situations occur perhaps once in 200,000 conversations, so a test with 108,000 exposed users has essentially no chance of observing one, and the online null is silence rather than evidence. Production decides the average outcome; a curated set retains authority over tail risks it was built to cover, because it is the only instrument that can see them. The correct move there is neither to ship nor to block on the aggregate — it is to fix the four cases and re-run.

The win is real and located somewhere you do not want it. If the entire 5-point lift comes from one segment while another segment is 3 points worse, the aggregate is a decision about who to serve, and that decision belongs to someone other than the experiment owner. Pre-registered segment breakdowns keep this from becoming an afterthought.

What the offline suite is still for#

Deciding that production settles this disagreement is not a demotion of the suite, and a team that reads it that way will stop maintaining the thing that makes online tests affordable. The suite runs in twenty minutes for about $9 of inference and can be pointed at any candidate on any afternoon. The online test costs fourteen days of calendar, a hundred thousand exposed users, and the opportunity cost of the surface being occupied by one experiment rather than another. Those are different instruments with different jobs, and using the expensive one for screening is how a team ends up running four experiments a quarter.

Three jobs stay with the suite regardless of what any online test says. It gates regressions in continuous integration, where the question is whether a change broke something that used to work and where a fast, deterministic answer on a fixed corpus is exactly right. It screens candidates before anything reaches shadow or interleaving, cheaply eliminating the ones that are obviously worse. And it holds the rare, severe cases that traffic cannot power, which is the fourth override condition stated as a positive.

The split team also needs handling, since the decision is being made in a room where half the people believe the suite. The productive move is to separate the ship decision from the credibility question and take both seriously: the challenger ships on Friday because the online test measured the outcome and the guardrails held, and the slice analysis runs the following week with the suite's maintainers doing it rather than having it done to them. A disagreement register that the same people fill in is an asset; one that a data scientist files against their work is a grievance.

Running this as a loop rather than an event#

Everything in this topic composes into a single path, and the part that most teams leave out is the arrow back.

The loop is what keeps the offline suite worth having. Without it the suite is a static artifact drifting away from a moving traffic distribution, and each drift is discovered by an expensive surprise. With it, every online test either confirms the proxy or improves it, and the cost of the improvement is a week of annotation on cases that real users generated.

The last box is the one to build next. Attaching evaluators to live production traces — LangSmith's online evaluation docs describe the mechanics, and Hamel Husain's account of the evals behind GitHub Copilot shows what it looks like when online signals gate every release — turns the quantities in this topic from things measured during an experiment into things measured continuously. Once the safety flag rate, the spend per conversation and a sampled rubric score run against live traffic all the time, the next experiment starts with a baseline that is already instrumented, and the fourteen days it takes to run are fourteen days of a system you can already see into.