Checking hourly is how a two-week test finishes on day three

At 14:00 on day three of a planned fourteen-day test, the primary metric's p-value reads 0.043. The product manager has the dashboard pinned in a browser tab and posts a screenshot in the channel with the message "we're significant, can we call it?" At 15:00 it reads 0.061. At 16:00, 0.038.

Nothing about the system changed between those three readings. The estimate is a random quantity that gets re-drawn every time more data arrives, and under a true null — the state of the world in which the two configurations really are identical and the observed gap is entirely sampling noise — it wanders. The experiment platform has recomputed the test 72 times since launch, and each recomputation was an opportunity for that wandering line to be below 0.05 at the moment someone looked.

The false-positive rate the dashboard is actually running at#

The 5% in "p < 0.05" is the probability that a single, pre-specified test rejects a true null. It says nothing about the probability that at least one of 72 sequential tests does. That quantity was worked out for clinical trials by Armitage and colleagues in 1969, and it is much larger than most people's intuition.

Seventy-two hourly looks puts this experiment somewhere around 34%. A third of changes that do nothing whatsoever will, at some point in a two-week window, show a "significant" result to someone refreshing the page, and if the team's rule is to ship on the first crossing then a third of the null changes ship. The rate is worse still if the crossing is allowed to end the test in either direction, and worse again because the looks are not independent draws — they are a random walk, so a crossing tends to persist for several hours, which makes it feel like a stable result rather than a moment.

The simulation is short enough to run against your own baseline, and it is the single most effective thing to put in front of a team that does not believe the table.

import numpy as np

rng = np.random.default_rng(0)
n_per_arm_per_hour, hours, sims = 420, 72, 5000
p_baseline = 0.62          # true completion rate, identical in both arms

crossed = 0
for _ in range(sims):
    a = rng.binomial(1, p_baseline, (hours, n_per_arm_per_hour))
    b = rng.binomial(1, p_baseline, (hours, n_per_arm_per_hour))
    ca, cb = np.cumsum(a.sum(1)), np.cumsum(b.sum(1))
    n = np.arange(1, hours + 1) * n_per_arm_per_hour
    pa, pb = ca / n, cb / n
    se = np.sqrt(pa * (1 - pa) / n + pb * (1 - pb) / n)
    z = np.abs(pa - pb) / se
    crossed += (z > 1.96).any()

print(f"false positive rate over {hours} looks: {crossed / sims:.1%}")
# false positive rate over 72 looks: 33.6%

Both arms are drawn from the same distribution. There is no effect to find. A third of the runs find one anyway.

Why the day-three winner is the worst one to ship#

Stopping at the first crossing does not merely raise the error rate; it biases the estimate. The moment the test stops is chosen because the observed difference is unusually large, so the reported effect is conditioned on being large, and the expected magnitude at a boundary crossing overshoots the truth. In the simulation above, the runs that crossed reported a mean absolute lift of 2.6 percentage points on a true effect of exactly zero. A team that ships those and then watches a post-launch dashboard sees the effect decay toward nothing over the following weeks and concludes the change "washed out", which is the same observation as "the effect was never there", described in a way that preserves everyone's belief in the day-three reading.

This also explains an asymmetry worth naming in the meeting. Nobody proposes ending the test early when the line dips the wrong way at 03:00; the dip is dismissed as noise, correctly, and the watching resumes. A stopping rule applied only in the favourable direction is not a rule.

Three defensible designs#

The problem is not that people look at the data. It is that a fixed-horizon p-value is only interpretable at the fixed horizon, and looking is a decision procedure. Three designs make continuous observation safe, and they trade off against each other.

Group sequential designs spend the total error budget across a small number of planned looks. Under an O'Brien–Fleming spending function the early boundaries are severe and the late ones are close to normal: with four looks, the first requires roughly p < 0.0001, the second p < 0.004, the third p < 0.019, and the final p < 0.043. That shape encodes the intuition correctly — a crossing at 25% of the data has to be extreme to be believed, and by the end the bar is nearly the ordinary one. Pocock's alternative uses a constant boundary, about p < 0.0158 for five looks, which stops earlier more often at the cost of a stiffer final bar.

Always-valid inference, popularised for online experiments by Johari, Pekelis and Walsh at Optimizely in 2015 and now standard in most commercial platforms, gives an interval that is simultaneously valid at every point in time. You can look every second and stop whenever you like, and the coverage guarantee holds. The price is width: early in a test the confidence sequence is wide enough that it will not detect anything, which is a feature, since that is exactly when a fixed-horizon interval is lying.

The stopping logic that comes out of this is a small state machine, and drawing it settles what "checking the dashboard" is allowed to mean.

The guardrail transition is the one that is allowed to fire on any look without a correction, and the reason is that the error costs are not symmetric. Stopping a harmful variant early costs a re-run; letting it continue costs users. Continuous monitoring of guardrails with an uncorrected threshold produces some false aborts, and false aborts are cheap. Continuous monitoring of the primary metric with an uncorrected threshold produces false ships, which are not.

What to say to the person watching the dashboard#

The product manager who posted the screenshot is not being unreasonable. They are looking at a number that the platform labelled "significant", and the label is the thing that is wrong. Three moves make that conversation productive rather than adversarial.

Show the A/A replay. Run the same dashboard against two arms that are both the current production configuration — many platforms will do this on live traffic — and let them watch a p-value that is guaranteed to be measuring nothing dip below 0.05 and come back. This converts an argument about statistics into an observation about their own dashboard, and it takes about ten minutes to set up.

Give them the decision they actually want, which is an earlier decision, and name its price. If shipping a week sooner is worth real money, a group sequential design with looks at days 4, 8 and 14 buys that legitimately for a few percent more traffic, and it has to be chosen before launch. Offering it now is offering to redesign the test around a result already seen.

Say what the current reading supports without overclaiming. At day three the confidence interval on this test runs from roughly −0.4 to +5.2 points. That is compatible with a large win, and it is also compatible with nothing, which is why the horizon exists. The change is not being blocked because someone dislikes it; the measurement has not finished.

Two practical questions this raises immediately#

The first is what to do about a test that is already running and has already been peeked at. There is no correction that repairs it afterwards, because the damage is in the decision procedure rather than in the data, and a p-value cannot be adjusted for looks that were not planned when the error budget was allocated. The two defensible options are to continue to the declared horizon and report the fixed-horizon result — the looks do not corrupt the data, only the licence to stop on them — or to switch prospectively to an always-valid procedure for the remainder, accepting that the switch itself was informed by having seen a favourable reading and reporting that fact alongside the result. What does not work is retrofitting a group sequential boundary onto looks that already happened and claiming the crossing was legitimate.

The second is what the team gives up by adopting a futility boundary, which is the part of sequential design that pays for itself most reliably and gets adopted least. Efficacy boundaries let a clear winner stop early; futility boundaries let a change that is obviously going nowhere stop early too. On a portfolio of experiments where most changes do nothing — which describes most portfolios, with published estimates from large experimentation programmes putting the share of changes with a measurable positive effect somewhere between 10% and 30% — futility stopping frees far more traffic than efficacy stopping does. A futility rule of "stop if the interval excludes the minimum detectable effect at the day-7 look" ends perhaps a third of tests a week early, and the freed capacity goes to the next candidate.

The dashboard itself is a design surface, and treating it as one removes most of these conversations before they start. A default view that shows sample accrual against the planned horizon, the guardrail panel with its declared limits, and a progress bar toward the analysis date — with the primary metric's estimate and interval present but marked as provisional, or hidden behind a click for tests that have chosen a fixed horizon — gives everyone the information they legitimately need while running. The provisional label matters more than the hiding, because people will find the number anyway, and a number that arrives with "this interval is not valid until day 14" attached is a different object from one that arrives with a green "significant" badge.

The corrections that do not apply here#

Peeking is one of three multiplicity problems that get conflated. Multiple looks at one metric is this chapter. Multiple metrics on one test was chapter 3's problem, and the fix there was a small guardrail set with operationally meaningful limits rather than an alpha correction. Multiple variants — four prompt candidates against one control — is a third, where a Bonferroni or Holm correction across the arms — both of which tighten the threshold each individual comparison must clear, so that the chance of any one of them firing falsely stays at 5% — is appropriate and where the sample size has to be divided among them.

Sequential methods also do not license stopping for arbitrary reasons. An always-valid interval protects the inference against when you look, not against choosing the metric, the segment or the population after seeing results. Stopping a test because the interval finally excluded zero on a segment you started slicing on Tuesday is not covered by any of this. The predeclaration from chapter 1 still holds, and the sequential design is an addition to it, specifying the looks and the boundaries alongside the metric, the unit and the exposure rule.

The last thing sequential analysis does not do is tell you how long the test should run when nothing stops it early. Boundaries need a horizon to spend the error budget against, and that horizon is not a number anyone should pick by feel — nor is it, as the next chapter shows, mostly a question about sample size.