Randomising per request puts one conversation in both arms

The experiment dashboard has been up for forty-eight hours and it looks like a win. Task completion is 62.4% in control and 66.7% in treatment, the difference is 4.3 percentage points, the p-value reads 0.008, and the confidence interval on the lift is [1.1, 7.5]. There are 243,000 requests in the test. The prompt change that produced this is a rewrite of the system message for a multi-turn support assistant, and the person who wrote it is asking, reasonably, whether they can ship on Monday.

What the dashboard is measuring#

The screen everyone is looking at is the readout of an A/B test: live traffic is split at random into two groups, called arms, one of them served by the configuration already in production, the control, and the other served by the candidate, the treatment. Nothing else about the two groups is chosen by anyone, so a systematic difference in what happens to them can be attributed to the configuration rather than to who happened to show up that week. The quantity the decision hangs on is the primary metric, task completion here, and the statistics beside it summarise how much of the 4.3-point gap ordinary luck could have produced: the p-value of 0.008 says a gap this large would appear less than once in a hundred repetitions if the two configurations behaved identically, and the interval [1.1, 7.5] is the range of true lifts the data is compatible with. The thing that gets flipped into an arm — on this test, an individual HTTP request — is the randomisation unit.

An A/B test is the load-bearing instrument of a wider practice, and the practice is worth naming before the test gets taken apart. Online evaluation is the measurement of a change by what real users do while the change is serving them in production: real traffic, real inputs, real consequences, over a window of calendar time. It sits opposite offline evaluation, which runs a candidate over a fixed collection of saved examples — a curated golden set, a public benchmark, a batch of traces harvested last month — and scores the outputs against a written rubric, by hand or with a model acting as judge. Offline evaluation is cheap, repeatable and available on any afternoon, because nobody is on the other end of it. Online evaluation costs days of calendar, exposes people to something that might be worse, and observes the one thing a saved dataset cannot contain, which is a person reading an answer and deciding what to do next.

The practice exists because the two kinds of number come apart routinely. A configuration can score higher on every example a team has written down and still resolve fewer tickets, because the saved examples are a proxy for the outcome and the proxy was assembled by people guessing at what real traffic looks like. A change can also improve the answers while making them slower, more expensive per conversation, or more likely to produce something the safety classifier objects to, none of which a rubric score reports. Online evaluation measures the outcome itself together with the costs incurred to reach it, and in a working system it arrives through four instruments that this topic covers in turn: A/B tests, interleaving, shadow deployment against mirrored traffic, and guardrail metrics. On the engineering side they show up as the experiment platform's dashboard, the guardrail panel beside it, a mirroring path in the gateway, and evaluators attached to live production traces. The Monday question is therefore a question about an online result, and 243,000 requests of real traffic are saying the rewrite works.

The answer is no, and the reason is not in any of those numbers. It is in the four lines of code that decided which arm each request went to.

# assignment.py — as deployed
def variant_for(request) -> str:
    bucket = int(hashlib.sha256(request.id.encode()).hexdigest(), 16) % 100
    return "treatment" if bucket < 50 else "control"

request.id is a fresh UUID minted by the gateway for every inbound HTTP call. A conversation with eight turns therefore makes eight independent coin flips, and the probability that all eight land in the same arm is 2 × 0.5**8, which is 0.8%. Over 99% of the multi-turn conversations in this test were served by both configurations, interleaved turn by turn.

What a mixed conversation actually does to the measurement#

Follow one session through the router and the problem stops being abstract.

Turn 3 is a control turn, and it is easy to resolve, because a treatment turn did the work at turn 2. Whatever task_completion is credited to at the end of that conversation, part of the causal chain runs through the other arm. Run the mirror image — treatment inherits a conversation that control has already muddled — and treatment gets penalised for control's turn. The two arms are contaminating each other in both directions, and the contamination is not noise that averages out, because it changes what each arm is being asked to do rather than just adding scatter around a true effect.

The formal statement of the assumption being violated is that a unit's outcome depends only on its own assignment, which experimental design calls the stable unit treatment value assumption, or SUTVA. It is the assumption almost every off-the-shelf analysis silently relies on. When it breaks, the estimate does not merely get less precise. It stops estimating the thing you wanted, which is the effect of serving the whole conversation with the new prompt. Nothing in this test ever served a whole conversation with the new prompt.

There is a second, quieter problem in those 243,000 requests. The unit of the primary metric is a conversation, since a task is either completed or not once, but the unit of randomisation is a request, and the analysis has been treating each request as an independent observation. Turns within a conversation are heavily correlated — a user who is going to give up gives up across all their turns — and on this feature the intra-cluster correlation for turn-level completion signals, which measures on a scale from 0 to 1 how strongly outcomes inside one conversation move together, sits around 0.35. The design effect for clustered data is approximately 1 + (m - 1) × ICC, where m is the average cluster size. With an average of 5.9 turns per conversation:

icc, m = 0.35, 5.9
design_effect = 1 + (m - 1) * icc          # 2.72
n_requests = 243_000
effective_n = n_requests / design_effect   # ~89,300
# the interval is sqrt(2.72) = 1.65x wider than reported

The [1.1, 7.5] interval is therefore closer to [-0.2, 8.8] once the correlation is accounted for, which crosses zero. The p-value of 0.008 was computed against a sample size the test does not have. Both defects push in the same direction — toward a confident-looking result — which is why per-request randomisation on a conversational product produces so many wins that evaporate after launch.

Choosing the unit, and what each choice costs#

The rule is to randomise at the coarsest unit at which a user could notice inconsistency, and no coarser. What counts as noticeable depends on the product, so the same change can justify different units in different places.

Power, in the column below, is the probability that a test of the size you can afford detects an effect of the size you care about, and it falls as the units get coarser because there are fewer of them.

Moving from request to conversation on this test drops the usable sample from 243,000 to about 41,000, and moving to user drops it again to roughly 26,000 because the average user in the window had 1.6 conversations. That is the honest cost. A cleanly randomised test on 26,000 users is a smaller test than the one on the dashboard, and it will take longer to reach a decision. It is also the only one of the three that answers the question the team asked.

The temptation to randomise finer is strong precisely because it works arithmetically: more units, tighter intervals, faster decisions, and a dashboard that never complains. Teams reach for it hardest on low-traffic features, where the coarse unit implies a horizon nobody wants to hear about, which is exactly where the resulting number is least trustworthy. If the correct unit leaves the test underpowered, the honest options are to run it longer, to pick a more sensitive metric, or to decide the change is not worth an experiment.

Catching it before the test, and the imbalance that survives a correct unit#

The mixed-conversation defect is visible thirty minutes after launch if anyone looks for it, which is the argument for a standing pre-flight check rather than a post-mortem. Group the assignment log by conversation, count the distinct arms per conversation, and report the share above one. On a correctly configured test that number is zero, and any value above it means the hash key is not the unit anyone agreed on. Running the same query as a hard gate — the experiment refuses to start accruing until thirty minutes of live traffic show a clean split — costs almost nothing and catches the class of bug where a refactor changes what unit_id is populated with two weeks after the design review.

A correct unit does not, on its own, produce comparable arms, and the way it fails is worth anticipating on any product with enterprise customers. Randomising by account on this feature means randomising 412 units, and those units are wildly unequal: the top 20 accounts generate 31% of all conversations, and the largest single account generates 6%. A fair coin over 412 units routinely produces arms whose volume differs by 8% or more, and since large accounts differ systematically from small ones — different task mix, trained users, custom knowledge bases — an imbalance in which giants landed where is an imbalance in the traffic mix, which confounds the comparison no matter how honest the coin was.

The fix is stratified assignment: sort units into strata by pre-period volume decile and randomise within each stratum, so each arm receives an equal number of giants, mid-size accounts and long-tail accounts. Where a single unit is large enough that no stratification helps — an account carrying 6% of traffic cannot be balanced against anything — the usual practice is to exclude it from the randomised population and evaluate it separately with its own owner, which is honest about the fact that one customer is not a sample.

There is a related bookkeeping point about standard errors. If the randomisation unit is the user but the metric is computed per conversation, the observations are still clustered, and the variance calculation has to account for it at the level randomisation happened rather than the level the rows are stored at. Most experiment platforms handle this if the unit is declared correctly; a hand-rolled analysis in a notebook usually does not, and it produces the same overconfident interval as the request-level test, arrived at by a different route.

Where per-request assignment is genuinely correct#

None of this makes request-level randomisation wrong in general. A document classification endpoint that takes one PDF and returns one label has no state, no continuity, and no user who could perceive two different behaviours, so each request is a legitimate independent unit and the design effect is 1. The same is true of an asynchronous enrichment pipeline, or a translation API called once per string by a build job. If the product has no memory, the unit that matches how it is experienced is the request.

The distinction is also not the same as the distinction between arms and rollouts. The gradual percentage ramps from the previous topic assign traffic by user or region for stability and blast-radius reasons, and they are excellent at telling you whether a build is safe to keep increasing. They are not designed to produce an unbiased estimate of an effect, because the population on the new build at 5% is not comparable to the population left on the old one, and the ramp schedule is confounded with the day of the week. A percentage rollout that showed no crashes is not evidence that the change helped anyone.

The randomisation unit is one of four things that have to be written down and agreed before the first request is served, and the reason for writing them down is that each of them can be adjusted afterwards in a way that manufactures a result:

• Primary metric. One metric, defined to the level of the SQL, with the exact aggregation. "Task completion" is not a metric; "share of conversations with a resolved status set within 24 hours of the last user turn" is.
• Randomisation unit. The unit, the hash key, and the salt. The salt matters because reusing the same hash key across experiments correlates their assignments and carries over any residual effect from the last test.
• Exposure rule. Which units enter the analysis at all, and by what logged event. This turns out to be the subtlest of the four and is the subject of the next chapter.
• Analysis window. The calendar span, and the rule for what happens to conversations that begin inside it and end outside.

Written down means in a document with a timestamp that predates the launch, not in someone's memory. Every one of these choices has a defensible version and a result-flattering version, and the difference between an experiment and a search for a favourable slice is whether the choice was made before the numbers existed.

The salt deserves a sentence of its own because it causes a specific, recurring failure. Two teams both hash user_id with no experiment-specific salt, so a user in the treatment arm of experiment A is in the treatment arm of experiment B with certainty. If either change has an effect, each test now measures the pair. Adding the experiment identifier to the hash input fixes it:

def variant_for(unit_id: str, experiment: str, treatment_share: float = 0.5) -> str:
    key = f"{experiment}:{unit_id}".encode()
    bucket = int(hashlib.sha256(key).hexdigest()[:8], 16) / 0xFFFFFFFF
    return "treatment" if bucket < treatment_share else "control"

variant_for(conversation.id, "support-sysprompt-v4")   # unit = conversation

The one-line change from request.id to conversation.id is the whole fix for the contamination, and it is worth noticing how small it is relative to how much of the result it invalidates. There is nothing sophisticated to build here. The failure survives because the wrong version produces a more encouraging dashboard and nobody is looking at the assignment code once the test is running.

Fixing the unit settles which conversations were served by which configuration. It does not settle which of them should appear in the analysis at all, and that question has its own way of driving an effect to zero: on this feature, most of the users who were assigned an arm never opened the assistant during the test window, and every one of them is currently in the denominator.