A rebuild that changed four things cannot credit the hierarchy

The last strategy in this topic is also the most expensive to adopt, and the evidence offered for it usually arrives in the same shape: a prototype, a benchmark, and a difference. That shape is where the interesting problem lives, because a hierarchical summary index cannot be built without also touching several other things. Summaries need leaves worth summarising, so the chunker changes. Everything is being re-embedded anyway, so the embedding model changes. The retrieval path has to handle more than one level, so it gets rewritten, and in the rewrite other parameters move.

What follows is therefore not really about summary trees. It is about what a difference between two systems measures when the two systems differ in more than one way, and it applies identically to a reranker proposal, an embedding upgrade or a chunking change. The quantity everybody wants is the contribution of one component. The quantity a prototype-against-production comparison produces is the contribution of every component that moved, added together, and no amount of care in running that comparison separates the terms.

The ladder that assigns the credit#

The comparison that isolates the hierarchy is flat chunks rebuilt with the same chunker, the same embedder, the same reranker and the same k — everything the branch changed except the tree. Running the intermediate arms as well is barely more work, because each is a config change over the same rebuilt index, and it tells you which of the four changes to keep if the hierarchy does not survive.

Read the last column as four gaps rather than five levels, since each row differs from the one above it in exactly one thing. The chunker is worth twenty-nine thousandths, the embedder nineteen, the deeper pool thirteen, and the tree is worth one thousandth in the wrong direction. Those four gaps add to sixty thousandths, which is exactly the distance from arm A to arm E and exactly the six points the proposal claimed. Nothing is left over, and the absence of a residual is what makes the decomposition usable: every point of the headline gain has been assigned to a row, so there is no unexplained remainder that the hierarchy might be hiding in.

This is the outcome to expect, and it is the outcome that shows up most of the time when the ladder gets run. The gain was real and it belonged to a better chunker, a better embedder and the retrieval depth from chapter 1, all of which are cheap, none of which requires a summarisation pipeline. The correct response to this table is to ship arms B through D immediately and put the tree back in the drawer.

The decomposition licenses less than it appears to, and the limit is worth stating before anybody quotes a middle row. The interval on a paired difference at this sample size, computed below, is about twenty-two thousandths. Only the chunker's twenty-nine thousandths clears that; the embedder's nineteen and the pool's thirteen do not. So the table supports "the hierarchy contributed nothing measurable" and "the chunker contributed something", and it does not support "the new embedder is worth nineteen thousandths of nDCG". Their sum is resolvable while the individual terms are not, which is the ordinary situation with small increments and is a reason to treat the ladder as a way of ruling things out rather than as a price list.

The arithmetic that makes any of this a fair comparison depends on one thing that is easy to get wrong. nDCG@10 is computed against per-passage relevance judgments, and the tree introduces retrievable units — document summaries, cluster summaries — that no judgment covers, because they did not exist when the set was labelled. Scoring an unjudged summary node as irrelevant penalises the tree for the labelling gap, and excluding it from the ranking flatters it. Both are wrong. The fix is to define relevance at the level of source spans rather than node IDs: a retrieved node counts as relevant if it derives from a judged-relevant span, which is checkable through the tree's child links, and to report end-to-end answer accuracy alongside, since that metric does not care what the units were.

import random
from statistics import mean

def paired_bootstrap(per_query_a, per_query_b, n=10_000, seed=0):
    """Per-query metric arrays, same query order. Returns the mean
    difference and a 95% interval on it."""
    rng = random.Random(seed)
    diffs = [b - a for a, b in zip(per_query_a, per_query_b)]
    idx = range(len(diffs))
    means = []
    for _ in range(n):
        sample = [diffs[rng.choice(idx)] for _ in idx]
        means.append(mean(sample))
    means.sort()
    return mean(diffs), (means[int(0.025 * n)], means[int(0.975 * n)])

delta, ci = paired_bootstrap(ndcg_arm_D, ndcg_arm_E)
print(f"hierarchy contribution: {delta:+.3f} (95% CI {ci[0]:+.3f} to {ci[1]:+.3f})")
# hierarchy contribution: -0.001 (95% CI -0.024 to +0.021)

The bootstrap resamples the per-query differences with replacement ten thousand times, which shows how far the mean difference could have wandered on a different sample of queries and therefore how much of the reported gap is luck. Pairing matters because the arms ran on identical queries, so the per-query difference removes the variance from queries that are simply hard. The printed interval runs from −0.024 to +0.021, whose midpoint is the reported −0.001 and whose half-width is a shade over twenty-two thousandths — which is where the ±0.022 resolution used above comes from. That figure sets the resolution of the whole exercise: a true effect of +0.01 is not measurable on this set at this size, and anyone reporting one is reporting noise. A +0.03 effect is measurable, and if you need to resolve smaller than that, the answer is more judged queries rather than more bootstrap samples.

Why the changes arrive bundled, and how to stop that#

Nobody bundles four changes out of carelessness. Re-embedding 1.9 million chunks is a two-hour job and $38 of API spend, the ingestion code has to be modified to emit a new index shape, and a prototype built to answer "is this idea worth pursuing" is naturally built as one branch. Running the ladder means four index builds instead of one, so roughly $150 — four times that $38, since each arm re-embeds the same chunks — and a day of wall-clock, plus the discipline to resist testing the good version first.

Set against that, the alternative is adopting a permanent summarisation pipeline on an experiment that never isolated its variable, and a day is cheap. The structural fix is to make the index shape a configuration value over a single ingestion path rather than a branch: same chunker code, same embedder client, same reranker, with index_shape: flat | tree and pool_depth as parameters. Arms then differ by config rather than by diff, which removes the possibility that some unrelated improvement rode along, and it makes the ladder reproducible six months later when somebody asks whether the conclusion still holds.

Keeping the leaf text byte-identical across arms is the other constraint worth enforcing, because it is what lets one set of relevance judgments serve all five rows. Once arm B changes the chunker, the leaves differ from production's and the judgments have to be re-mapped by span overlap — which is doable and is a place errors enter quietly. The cleanest sequencing is to fix the chunker first, re-judge or re-map once, and then hold it constant while the shape question is settled.

Where an online A/B is the right instrument#

A clean +3 on the stratified offline set is still a claim about 400 judged queries drawn from a traffic sample, and the honest scope of that claim is "on queries like these, retrieval improves". Production traffic drifts, contains a heavy head of repeated questions the eval set deliberately de-duplicated, and includes malformed queries nobody labelled.

The instinct to settle it with an online A/B is right in principle and usually underpowered in practice. If the observable is a thumbs-down rate sitting at 12%, detecting a two-point absolute change at 80% power — an eight-in-ten chance of seeing the effect at all, supposing it is really there — needs on the order of 8,000 sessions per arm, and most internal assistants do not serve that in a month. Explicit feedback is also sparse and biased toward the annoyed.

That does not make online measurement useless; it makes it an instrument for a different question. Offline evaluation cannot see p95 latency after traversal, cost per query, the rate of prompts composed mostly of summaries, or complaints that the assistant is describing a document as it was yesterday — and none of those four appear in nDCG at any sample size, because the judged set has no opinion about them. Ship behind a flag, watch those four, and keep the accuracy question where the judgments are.

Suppose it survives at +3#

If arm E comes in at 0.711 against arm D's 0.681, with an interval that excludes zero, the hierarchy is doing something and the decision becomes a cost question rather than an evidence question. What you would be adopting is not an index configuration but a standing system: the daily summarisation job from the previous chapter with its own alerting and its own backfill procedure, a level quota in the retrieval path that someone has to maintain, 30 to 60 ms of added query latency if you use traversal, and a staleness window that becomes a property of the product — the assistant's answers about a document can lag its edits by up to the regeneration interval, which is a sentence that has to appear in the documentation.

There is also an evaluation obligation that outlives the decision. A summary index has a failure mode the flat one does not, where paraphrase outranks primary text on specific-fact queries, and it does not announce itself; it shows up as a slow drift in accuracy on exactly the queries nobody stratifies for. Shipping the tree means adding the level-composition metric to the dashboard and keeping the stratified eval running, which is ongoing work rather than a launch task.

Three points of nDCG is worth all of that in some systems and not in others. The question that decides it is whether the queries the tree helps are queries the product is judged on, which is not answerable from an aggregate at all.

Testing the claim the proposal is actually making#

The argument attached to these proposals is usually specific: summaries help most on broad questions like "what changed in Q3". That is a testable claim on its own, separate from the aggregate, and it has clean falsifiers.

Label the judged set into strata before looking at any results — aggregative questions whose answer requires synthesising across documents, and specific-fact questions answered by one passage — and have two people label independently so the boundary is not drawn by whoever wants the result. Four hundred queries typically split around 15/85, which is itself informative and often the end of the discussion. Fifteen per cent of 400 is a 60-query aggregative stratum, and an interval scales as one over the square root of the sample, so shrinking from 400 queries to 60 widens ±0.022 by the square root of the ratio between them — a little over two and a half times, which lands at the ±0.055 that stratum can resolve and nothing smaller. Then compare arm D against arm E within each stratum.

Weight each row by the 15/85 split and something uncomfortable falls out. The claim-holds row aggregates to fifteen hundredths of +0.09, which is a little over one hundredth overall — below the ±0.022 the pooled set can resolve. The evenly-spread row aggregates to +0.03 and the inverted row to a shade over three and a half hundredths, both of which the pooled set can see. So the two outcomes visible in an aggregate are precisely the two that falsify the proposal's story, and the outcome that confirms it is invisible there. A team that runs only the pooled comparison will therefore see a movement exactly when the tree is not doing what it claims, which is a strong argument for stratifying before the run rather than after a disappointing one.

The even split is the most instructive falsifier, because it says the effect is not coming from the mechanism it is named after. If summaries genuinely supply what no leaf contains, the benefit has to be concentrated where no leaf contains the answer, and a uniform lift means arm E and arm D differ in some other way — usually the summary text acting as an extra topical field that improves matching generally, which is a real effect and one you can get far more cheaply by prepending headings or generated titles to leaf chunks.

The inverted outcome deserves investigation rather than dismissal. Summaries improving specific-fact retrieval usually means the summary is functioning as a router, matching a query and dragging its correct leaf children into the prompt — which is a genuine benefit, and it means the thing to build is a document-title or keyword field on the leaves, not a three-level tree.

The order to build in#

The order these chapters were written in is not the order to build in. Establish a labelled set and a harness first, since every technique here is a conditional improvement whose condition is a property of your corpus, and none of them can be evaluated without one. Measure the baseline honestly, including the k sweep from chapter 1, because a tuned flat pipeline is the thing every later strategy has to beat and it is frequently within two points of the sophisticated one. Then change one variable, keep the arm, and record the config hash next to the number so the next person knows what the number was measured on.

The strategies themselves sort by what they cost to run rather than by how advanced they sound: k and rerank depth are free, diversity is nearly free, expansion costs tokens, merging costs a threshold to maintain, and summary trees cost a pipeline. Reach in that order, stop when the measurement stops moving, and treat a technique you cannot show a stratified gain for as a technique you have not adopted yet.

What the ladder is really enforcing is a rule about what a number is allowed to be about. A difference between two systems is a statement about the pair, and it becomes a statement about a component only when the pair differs in that component alone. Every arm you skip is a term you have folded into the total without noticing, and the folding is invisible afterwards — the reported figure looks exactly the same whether it is one effect or four. That is why the extra index builds are cheap at almost any price: they are not buying precision, they are buying the right to name a cause.