Online evaluation

Read these first

• Release and rollout

What you will know

• Run an A/B test on a model or prompt change

  Use a predeclared primary metric, randomization unit, exposure rule, and analysis window; otherwise traffic mix can confound the result.

• Choose a guardrail metric that stops a harmful winner

  Set explicit thresholds for failure modes such as complaints, unsafe outputs, latency, and spend, with a clear abort owner.

• Shadow traffic against a new configuration before switching

  Replay requests without serving the responses, while preventing side effects, sensitive-data leakage, and unexpected dependency load.

• Decide how long to run before calling it

  Account for sample size, variance, novelty effects, and sequential peeking rather than stopping after an arbitrary number of hours.

• Compare offline and online rankings for the same change and document a disagreement

  Production traffic adds user-mix shifts, feedback behavior, tail latency, and abuse patterns that static datasets usually omit.

Chapters

1. 1

   Randomising per request puts one conversation in both arms

   Two days in, treatment leads on task completion with a comfortable p-value, and nobody should act on it: assigning each incoming request independently means a control turn answers a question shaped by a treatment turn, so the arms are not independent and the interval is narrower than the evidence supports. Pick the randomisation unit that matches how the product is experienced, then live with the power that leaves you.

   Read chapter
2. 2

   The analysis starts at exposure, the randomisation stays at login

   Users are split at login, only about 30% ever open the AI feature, and an analysis over everyone assigned reports no effect. The 70% who could not experience either arm dilute the estimate toward zero. Move the analysis population to the moment of exposure, logged by an identical rule on both sides — logging exposure only where the treatment fires reintroduces exactly the bias randomisation removed.

   Read chapter
3. 3

   A guardrail is a limit declared before the winner is known

   Treatment wins resolution rate by 4 points while spend per conversation rises 35%, p95 latency 900ms, and safety-flagged responses go from 0.2% to 0.45%. Deciding how to weigh that once the numbers exist is how a guardrail becomes a trade. Set thresholds and a named owner authorised to abort before launch, and understand why a relative doubling is not excused by a small absolute rate.

   Read chapter
4. 4

   Checking hourly is how a two-week test finishes on day three

   Recomputing significance every hour for three days is dozens of looks, each a chance to cross p < 0.05 under a true null. Shipping on the first crossing selects the moment noise was largest, which is why such winners fail to reproduce after launch. Covers fixed horizons, sequential tests built for continuous monitoring, and the conversation with the product manager watching the same dashboard and reading the same dip.

   Read chapter
5. 5

   Power gives you one day; the week and the novelty give you the rest

   At 20,000 sessions a day split evenly, detecting 2 points on a 62% baseline needs roughly 9,000 per arm — about a day of traffic — and a horizon set from that alone is indefensible. Three more inputs fix the real number: whole weekly cycles, because weekday and weekend traffic are different populations; novelty decay, because a visible change flatters itself most in its first days and a result that landed inside them is the part you cannot keep; and the lag between exposure and the outcome being measured. Randomising by conversation or user rather than by request costs power as well, pushing the same figure outward again.

   Read chapter
6. 6

   Interleaving measures preference, not the effect of shipping

   Merging two rankers' results into a single list per query compares them within a query instead of between users, which is far more sensitive per query — and measures a different quantity: relative preference between two lists. Nothing in an interleaving result says what happens to task completion, spend, or tail latency once one ranker serves alone, so it informs which A/B to run rather than replacing it.

   Read chapter
7. 7

   Shadow traffic cannot observe anyone reacting to the answer

   A week of shadowing shows the candidate beating production on 68% of sampled requests by your offline rubric, and the team proposes skipping the test. No user saw those outputs, so acceptance, retries, escalation, and abandonment left no trace — the primary metric measures a reaction shadowing cannot produce. What it does buy is behaviour on realistic inputs, without side effects or sensitive data leaking, before anyone is exposed.

   Read chapter
8. 8

   When the golden set and production disagree, production decides

   A trusted offline suite ranks the incumbent eight points above a challenger; the online test gives the challenger a clear win with no guardrail breached. A track record shows the suite predicted well on the changes it has seen, not that it covers this one, and the online test measures the outcome the suite was only ever a proxy for. Mine the disagreement for the slices nobody wrote cases for, and record it.

   Read chapter