Interleaving measures preference, not the effect of shipping

There are two candidate retrieval rankers behind a support assistant, and the surface they feed shows the user a list of source passages alongside the generated answer. The feature handles about 8,000 queries a day. The click-through rate on the passage list is 24%, and the plausible improvement from the new ranker is around one percentage point.

A user-randomised A/B on that needs roughly 29,000 queries per arm for 80% power, which at 4,000 queries per arm per day is seven days of pure traffic, and once the clustering of queries within users and the two-week weekly-cycle constraint are applied it lands at three weeks. Three weeks to choose between two rankers, when there are five more candidates behind these two, is a research programme rather than an experiment.

The alternative is to stop comparing users and start comparing lists inside a single query.

One list, two sources, per-query attribution#

Interleaving merges the outputs of both rankers into one list, shows that list to the user, and records which ranker contributed each item the user clicked. Every query becomes a paired comparison, so the between-user variance that dominates an A/B — some people click a lot, some never click — cancels out entirely.

The naive merge — alternate items from each list, skipping duplicates — is called balanced interleaving and it has a bias that shows up exactly when the two rankers are similar. If ranker B is ranker A with one extra document inserted at the top, balanced interleaving systematically credits clicks to B on positions where either ranker would have put the same document, and B wins comparisons it did not earn. Radlinski, Kurup and Joachims described the fix in 2008 as team-draft interleaving, which borrows the mechanics of picking teams in a playground: the two rankers take turns choosing their most-preferred remaining document, coin-flipping for who picks first in each round, and each document is permanently attributed to the team that drafted it.

import random

def team_draft(list_a, list_b, k=8):
    """Returns (merged, attribution) where attribution[doc] is 'A' or 'B'."""
    merged, attribution = [], {}
    ia = ib = 0
    while len(merged) < k and (ia < len(list_a) or ib < len(list_b)):
        a_first = random.random() < 0.5
        for team, lst, idx in (("A", list_a, "ia"), ("B", list_b, "ib")) if a_first \
                          else (("B", list_b, "ib"), ("A", list_a, "ia")):
            i = ia if idx == "ia" else ib
            while i < len(lst) and lst[i] in attribution:
                i += 1
            if i < len(lst) and len(merged) < k:
                merged.append(lst[i])
                attribution[lst[i]] = team
                i += 1
            if idx == "ia":
                ia = i
            else:
                ib = i
    return merged, attribution

Scoring is a count per query: the ranker whose documents received more clicks wins that query, and queries with equal credit are ties. Across queries the statistic is a sign test on the wins, which asks only whether one side wins more queries than the other and discards the magnitudes, and that is why the sample requirement collapses. To detect a 55:45 preference split at 80% power needs about 780 queries in which a preference is expressed, and since roughly 30% of queries here produce at least one click, that is about 2,600 queries — eight hours of traffic rather than three weeks.

from math import sqrt
from scipy.stats import norm

p = 0.55  # share of decisive queries won by B
n = ((norm.ppf(0.975) * 0.5 + norm.ppf(0.8) * sqrt(p * (1 - p))) / (p - 0.5)) ** 2
print(round(n))                      # 782 decisive queries
print(round(n / 0.30))               # 2607 queries total, ≈ 8 hours

Chapelle and colleagues validated this at scale in 2012 across Yahoo, Bing and Yandex data, finding that interleaving needed one to two orders of magnitude less data than A/B tests to reach the same confidence about which ranker was preferred, and that the direction of the interleaving verdict agreed with the A/B verdict on most pairs they could compare both ways.

The quantity it estimates is not the quantity you ship on#

The agreement in direction is the part people quote. The part that matters more for a shipping decision is that the magnitudes did not correspond: knowing that B wins 56% of decisive queries tells you nothing about how much task completion moves when B serves every query alone.

Three things make interleaving structurally unable to answer that. The list the user saw was produced by neither ranker, so the presentation is a third ranking whose position effects, diversity and redundancy differ from either candidate's real output. The comparison is relative, and both rankers can be poor — a 60:40 preference between two configurations that each fail the user most of the time is a real result about their ordering and no result at all about whether either should ship. And the outcome recorded is a click on a passage, which is the immediate, attributable signal that makes attribution possible, and which is not the primary metric anyone declared in chapter 1.

The guardrail row is the one that ends the argument in practice. Interleaving requires running both rankers on every query, so the serving cost is the sum of both and the latency is the maximum of both, which means the run cannot observe the cost or latency profile of either ranker serving alone. Chapter 3's spend and p95 limits are unmeasurable in an interleaved run by construction.

The merged list is something a user has to live with#

An interleaved run is not an unobtrusive measurement. Every query in it is answered with a list that neither candidate would have produced, and that list can be worse than both. Team draft alternates provenance, so a query where ranker A's top three documents are the correct chain of reasoning gets those three split across positions 1, 3 and 5 with B's material in between, and a user scanning the first two passages may leave with half the answer. On a surface where the retrieved passages feed a generated answer, the degradation is worse still, because the generator receives a mixed context and produces a response neither ranker would have supported.

That makes an interleaving run a live experiment on real users with its own risk profile, and it deserves the guardrails from chapter 3 even though it cannot measure most of them. The practical set is a cap on exposure — 5% of queries interleaved rather than all of them, which still delivers 2,600 decisive queries in a few days — a check that the interleaved population's task completion has not fallen relative to the non-interleaved remainder, and a kill switch on the merge. Teams that treat interleaving as measurement rather than as a deployment are the ones surprised when a week of it shows up as a dip in the weekly metrics review.

Scoring has more edges than the description suggests. A query where the user clicks nothing is a tie and contributes no information, which is why the traffic requirement is three times the decisive-query count. A query where both teams' documents are clicked equally is also a tie. Documents that both rankers place first are drafted by whichever team won that round's coin flip, so the pair's shared core contributes noise rather than signal, and a comparison between two rankers that agree on 90% of their top-8 sets is decided almost entirely by a handful of differing positions. Reporting the overlap alongside the verdict keeps that visible.

Extending beyond two candidates is possible and is the reason the technique scales as a screening tool. Multileaving, introduced by Schuth and colleagues in 2014, drafts from an arbitrary number of rankers into one list and attributes clicks the same way, so six candidates can be compared in a single run rather than in fifteen pairwise ones. The cost is that each ranker contributes fewer positions per query, so the per-query information falls and the traffic requirement rises, though nowhere near as fast as running the pairs separately would.

What it is worth using for#

The right shape here is to interleave first and A/B once. Interleaving over a day sorts seven candidates into an order and eliminates the ones that lose to the incumbent, which is a screening job it does at roughly a hundredth of the traffic cost. The survivor then goes into a single A/B against production, with the primary metric, the exposure rule, the guardrails and the fourteen-day horizon from the preceding chapters, and that test answers the shipping question including everything interleaving could not see.

Skipping the A/B saves three weeks and gives up the estimate of the effect, which is the number that the business case, the guardrail check and the post-launch monitoring all consume. Skipping the interleaving costs eighteen weeks of sequential A/B tests to screen seven candidates, most of them on candidates that a day of interleaving would have eliminated.

There is also a class of change interleaving simply cannot be applied to. It needs an output that is a list of independently attributable items with per-item feedback. A single generated answer cannot be interleaved, because there is no way to merge two paragraphs into one paragraph and attribute the user's satisfaction to half of it. Changes to the system prompt, the generation model, the temperature or the answer format are all outside its scope, and reaching for it there produces either a side-by-side presentation — which is a survey, not the product — or nothing.

For those changes the sensitivity trick that survives is a different one: paired offline preference, where a human, or an LLM judge — a language model prompted to compare two outputs against written criteria and pick one — sees both outputs for the same input. That recovers the within-input pairing and loses the real user entirely, which is a different trade and belongs to the offline suite rather than to online evaluation.

The bias worth knowing about before trusting it#

Click-based interleaving measures what gets clicked, and the systematic difference between what gets clicked and what helps is well documented in the ranking literature. A ranker that surfaces passages with alarming titles, or that duplicates the same document with three different anchors, will win interleaving comparisons and serve the user worse. Attribution also gets murky when both rankers return overlapping documents, since team draft assigns each duplicate to whichever team drafted it first, and a document both rankers ranked first is credited to a coin flip.

The mitigation is to score on something closer to satisfaction than a raw click — a click followed by no reformulated query within 60 seconds, say, or a click on a passage that the generated answer went on to cite — and to report the interleaving result alongside the share of queries where the two rankers returned substantially the same set, because a 55:45 verdict on a pair that agrees on 90% of documents is a verdict about very little.

None of the designs so far can be run without exposing users to the candidate. Interleaving shows them a merged list, an A/B shows half of them the new configuration, and both require a decision that the candidate is safe enough to serve. There is a way to observe a candidate on real production inputs before anyone is exposed to it at all, and it comes with its own hard limit.