A nine-point win with three variables moved measures nothing

A chunking comparison exists to answer one question: does moving the boundaries between chunks change how often a retrieval system finds the right text. The answer is a difference between two numbers, and a difference is only interpretable when everything except the boundaries has been held still. Confounding is the name for what goes wrong otherwise — two or more causes changing together, so the effect that gets measured is their sum and no part of it can be assigned to any one of them. It is the ordinary condition of a retrieval experiment rather than an exotic failure, because the components of a retrieval system are all cheap to change and all move the same headline number.

The stakes are not academic. A chunking migration means re-parsing, re-chunking, re-embedding and re-indexing a whole document collection, and the decision to do that gets made on a single figure posted in a channel. An attributable difference is the minimum that figure has to be, and attribution alone is not enough — a comparison can be perfectly controlled and still be scored on a metric that cannot see the variable it controlled.

Nine points of what?#

Three segmentations of one 12,000-document support corpus, evaluated against the same 140-query set and scored on recall@10 — the fraction of queries for which at least one chunk marked relevant appeared among the ten the retriever returned — come back like this:

The headline is the distance from C to the best of the others, which is 0.70 against 0.61, or nine points on a scale where one point is a hundredth. That is a large enough gap to make a decision on. It is also a gap that run C did not earn alone: run C was executed a week after A and B, on a branch that had also switched the retriever from k=8 to k=20 before the cross-encoder rerank — the stage where a model that reads the query and a candidate chunk together, rather than comparing two vectors, rescores the candidates and reorders them — and had re-embedded with bge-base-en-v1.5 where A and B used bge-base-en-v1.0.

Three things differ between run C and run B: the boundaries, the retrieval depth feeding the reranker, and the embedding model. The measured quantity is the sum of their effects plus noise, and there is no way to attribute any of it. That is not a pedantic objection about experimental hygiene; the individual effects here are plausibly large enough to account for the entire gap on their own.

Raising retrieval depth from 8 to 20 candidates before a cross-encoder rerank changes recall@10 directly, because the reranker can only promote documents the retriever handed it. If the correct chunk sat at rank 12 for some queries, k=8 guaranteed a miss and k=20 gives the reranker a chance to pull it into the top 10. On a corpus with this profile that adjustment is typically worth three to five points on its own, and it costs about 40 ms of extra cross-encoder latency per query, which is the reason it was not already at 20.

Each link in that chain is checkable on its own, which is what makes this confound first-order rather than a rounding error. A reranker takes a candidate list and returns the same list in a new order, so nothing enters the output that was not already in the input. A chunk sitting at rank 12 is outside a list of eight and inside a list of twenty. Recall@10 asks whether a relevant chunk appears among the first ten of the reordered list, and a chunk that never entered the list cannot appear among them at any position. Changing the depth therefore changes which outcomes recall@10 is capable of reporting, before any question of which segmentation is better has been asked.

The reason k had been left at 8 names a distinction worth keeping visible, because its two halves behave differently under scrutiny. The latency budget is a requirement: somebody decided what a query is allowed to cost, and that decision sits outside this experiment. The value of k is a design decision taken underneath that requirement, and it is the kind of thing an engineer changes on a branch without thinking of it as a variable at all. A confound almost never arrives labelled as one; it arrives as a setting somebody improved.

The embedding model change is harder to bound but no less real. The two bge-base-en releases differ by roughly two points of average retrieval score on MTEB, the public Massive Text Embedding Benchmark that most embedding models report against, and the gap on any specific corpus can be double that or can be negative, since a model that improved on average did so by improving on some domains and regressing on others. Two to six points is a fair range to assume for an unmeasured swap.

The reason a benchmark average cannot be carried across is worth stating rather than assumed. An average over many domains is a statement about a population of corpora, and your corpus is one draw from something that may not even be that population. A model that gained on the benchmark by improving question-answering over web text has no obligation to improve retrieval over vendor support documents, and the mechanism by which it gained — different training data, different negatives, a different pooling step — is exactly the mechanism that could cost it points on a domain the benchmark under-weights. The benchmark delta bounds nothing about a single corpus; it only says what size of effect is worth taking seriously.

Read that table by the gaps between its rows rather than by the numbers in them. The middle two rows have ranges because somebody could estimate them from outside this experiment. The top row has no range at all, and its emptiness is the finding: the one quantity the experiment was run to obtain is the only one the experiment cannot supply. The bottom row is a different kind of entry again — it is not an effect, it is the width of the instrument, and it applies to every other row whether or not anything was confounded.

The residual can be computed, and doing it explicitly is what turns the table into an argument. The measured gap is the boundary effect plus the depth effect plus the model effect, so the boundary effect is the gap with the other two subtracted off. Take the corner most favourable to segmentation: the depth change worth only its smallest estimate, and the model swap actually costing a point rather than gaining one. Nine minus three is six, and subtracting a negative one adds it back, which leaves seven. Now take the corner least favourable: the depth change at its largest and the model swap at its largest. Nine minus five is four, and four minus six is minus two. So before any allowance for noise, the boundary effect lies somewhere between minus two points and plus seven.

Add the middle rows and the confidence band, and the residual attributable to heading-aware splitting spans something like minus three points to plus nine. The honest report of this experiment is that the effect of segmentation on this corpus is unknown, and it is unknown in the specific sense that the experiment could not have distinguished a strong positive effect from a mild negative one.

The gap between the corner arithmetic and the span just quoted is the sampling band, applied outward at both ends and rounded rather than carried to a decimal. That rounding is deliberate. Both component ranges are estimates borrowed from outside this experiment, and stating a residual to a decimal place would imply a precision that neither of them has.

The sampling-noise row deserves its own attention, because it applies even to a perfectly clean experiment. A 140-query evaluation set measuring a proportion around 0.6 has a standard error near 4 points on each arm. Comparing two arms unpaired would make a nine-point difference barely significant. The comparison is paired, though — the same queries run against both indexes — and pairing removes most of the variance, because a query that is hard for structural reasons is hard in both arms. A paired bootstrap over the query set, resampling queries with replacement 10,000 times and recording the difference each time, is about six lines of code and turns "B beat A by three points" into "B beat A by three points, 95 percent interval −1 to +7", which is a different sentence to take into a migration meeting.

Neither of those two claims is a convention, and both come out of one line of arithmetic each. For the standard error: the variance of a proportion is the proportion multiplied by one minus the proportion, divided by the number of observations, and the standard error is the square root of that. Six tenths times four tenths is a little under a quarter; dividing that by 140 queries gives a small fraction; taking the square root of the fraction lands just above four hundredths, which is the four points. The shape of the expression is worth holding on to, because the query count sits under a square root — to halve the standard error you need four times as many queries, so buying your way out of a noisy evaluation is expensive in a specific and predictable way.

Pairing earns its variance reduction by subtraction. A query whose answer was never parsed correctly contributes a miss to both arms, so it moves both numbers by the same amount and moves the difference between them not at all. The same holds for a query that is trivially easy in every configuration. What survives in the paired difference is only the queries the two arms disagree about, and there are far fewer of those than there are queries — which is why the interval around a paired difference is narrower than the intervals around the two numbers being differenced, a result that looks wrong until the cancellation is visible.

Pinning the things that are not the experiment#

The repair is mechanical. Fix everything except the boundaries, and make the fixture refuse to run if anything else has moved.

Everything downstream of the three splitters is one object with one hash. The splitters are the only fork in the graph.

That is a claim about shape rather than a picture of a build, and it is what makes the drawing worth having. A single parsed corpus enters at the left, the graph forks exactly once into three splitters, and it rejoins immediately into one embedding step, one index and one evaluator. Any confound is a second fork somewhere in that path, and a second fork shows up here as an extra arrow, which is what makes the drawing checkable against a run manifest rather than decorative.

import hashlib, json

PINNED = ("parse_sha", "embed_model", "embed_dim", "metric",
          "hnsw_m", "hnsw_ef_construction", "retrieve_k",
          "reranker", "rerank_to", "query_set_sha", "evaluator_version")

def pin_hash(cfg: dict) -> str:
    subset = {k: cfg[k] for k in PINNED}
    return hashlib.sha256(json.dumps(subset, sort_keys=True).encode()).hexdigest()[:12]

def compare(runs: list[dict]) -> None:
    hashes = {pin_hash(r["config"]) for r in runs}
    if len(hashes) > 1:
        raise ValueError(
            f"runs differ outside the segmentation variable: {sorted(hashes)}")
    for r in sorted(runs, key=lambda r: -r["recall_at_10"]):
        print(f'{r["config"]["splitter"]:<24} {r["recall_at_10"]:.3f}')

The mechanism in that code is worth naming, because it is doing something slightly different from what a reader expects a hash to do. pin_hash does not hash the whole configuration; it selects the fields listed in PINNED and hashes only those, sorting the keys first so that two configurations written in a different order produce the same digest. The splitter's own name is deliberately absent from PINNED, which is what makes the check possible: because the one field that is meant to differ has been excluded, two legitimate runs collapse to the same digest, and any other difference — a changed retrieve_k, a re-embedded corpus, a query file edited after the fact — makes the digests differ. Collecting the digests into a set and testing whether it has more than one member is therefore an exact test for "something other than the splitter moved". The set has one member when the comparison is valid and more than one when it is not, with no in-between state and nothing to tune.

Running that against the results above raises immediately, which is the point of it. A comparison function that cannot produce a number when the comparison is invalid is worth more than a dashboard that always produces one. The cost of this discipline is real and should be stated plainly: on a 90,000-document knowledge base, re-embedding for one configuration runs about four hours of GPU time and takes an engineer roughly a day to prepare, validate and load. Three configurations is twelve GPU-hours and three engineer-days, and re-running two of them because the third moved a variable doubles that. The temptation to accept the original number is a budget pressure rather than a lapse in rigour, which is why the guard belongs in code where nobody has to volunteer for it.

The answers to refuse#

A pinned experiment can still hand you a number that does not mean what the meeting will take it to mean, and recall@10 is the standing example. It asks whether any of the ten retrieved chunks came from the region of the corpus labelled relevant for this query, and that is a coarser question than the pipeline actually faces. A chunk that says "Claims must be filed within the window defined in section 4.2" comes from the right document, sits in the right section, and is labelled relevant by any judgement based on document identity or on section identity. It also does not contain the answer.

This is where the tempting belief lives, and it is tempting because recall@10 is a real metric that really does detect real failures. A team that has watched it catch a broken index reasonably concludes that it is a working defence. What makes it fail is that it stays fixed while the thing it has to discriminate moves: the label it checks against is document or section identity, and the quantity under test in a chunking experiment is where inside a section the boundaries fall. Change the boundaries and the section labels do not change at all. A defence that cannot vary on the axis being tested will report the same verdict whatever happens along it, and it will do so while continuing to catch the unrelated failures that earned it its reputation.

The metric to insist on alongside recall is answer-span containment: for each query, the annotator marks the exact span of text that answers it, and the metric asks whether any retrieved chunk contains that span in full. It is more work to build — 140 queries with marked spans is perhaps a day and a half of careful annotation — and it separates the two cases that recall conflates.

That table enumerates every way a retrieval can turn out, and it closes with nothing left over, which is what makes it more than a list of examples. Read the two metric columns against each other and they agree on exactly two rows: the top one, where the retrieved chunk holds the answer text, and the bottom one, where nothing relevant came back at all. On the three rows in between they disagree completely — recall calls all three a hit and containment calls all three a miss. Since those are the only rows left once the two extremes are accounted for, the disagreement between the two metrics is not spread thinly across the space of outcomes; it is concentrated entirely in the middle of it.

The three middle rows are exactly the failure modes that chunking changes, so a chunking experiment scored only on recall@10 is scored on a metric that is partly blind to the thing under test. In the heading-aware run above, when the corpus was re-evaluated with spans marked, recall@10 held at 0.70 while span containment came in at 0.52 — because heading-aware sections on this corpus are long, and a long section that mentions the topic satisfies recall while frequently placing the answer sentence past the encoder's input limit. Recall was measuring the section; containment was measuring the sentence.

What the table licenses is a sharper statement than "recall is noisy". Recall@10 is not an imprecise estimate of containment, and no sample size makes it converge on one; it is a different question that happens to return the same verdict at the two extremes. That is why a team can validate it against an obviously good run and an obviously broken one and see nothing wrong with it. The pair of numbers above is those three middle rows appearing as a gap between two columns, and the size of that gap is a property of how long the sections are, not of how good the retrieval is.

The query set itself has to be pinned as hard as the index parameters, and it is the component most likely to drift without anyone deciding to change it. Write the queries against the corpus before you have seen a single result, sample them from real logs rather than inventing them, and freeze the file under a hash the harness checks. A query set that gets "cleaned up" between run B and run C — three ambiguous questions dropped, two rephrased because they seemed unfair — is a moved variable with no entry in the manifest, and its effect is unbounded because dropping the hardest three queries from a 140-query set moves the mean by two points on its own.

That last figure is arithmetic rather than an estimate, and it is worth doing because it shows how little has to move. Three queries out of 140 is a little over two hundredths of the set. Removing three queries that were certain misses takes three failures out of the numerator's complement and three items out of the denominator, and the fraction shifts by very close to that same two hundredths — which, on a scale where one point is a hundredth, is the two points. No result changed; only the set the result was averaged over.

There is a second class of answer to refuse, which is any single aggregate over a query set that mixes question types. If 30 percent of your queries are one-sentence lookups and 70 percent are procedural "how do I" questions, a chunking change that helps one class and hurts the other nets out to a small positive and gets shipped, and the procedural users get worse service with no signal anywhere. Reporting the metric split by query class costs nothing once the classes are labelled, and it is the difference between "recall went up three points" and "recall went up eight points on lookups and down two on procedures".

A third answer worth refusing is a retrieval win reported without its ingest cost, on the grounds that ingest is a batch job nobody watches. If a configuration raises recall@10 by three points and p95 ingest time — the 95th percentile of per-document ingest latency, so the slow tail rather than the average — by 40 percent, the three points are a one-off gain and the 40 percent is a permanent tax. It is charged again on the next embedding-model upgrade, on every re-parse after an extractor fix, and on every future configuration you wanted to compare — which is to say it raises the price of the experiment you just ran, every time you want to run it again. Whether that trade is worth taking depends on what the extra time is buying, and chapter 6 works through the case where it is a structural property of the strategy rather than an implementation detail someone can optimise away.

Attribution is not coverage#

Pinning gets you an attributable difference between three configurations you chose. It does not tell you that one of them is good, and it does not search the space — three points on a grid, none of them near the optimum, will still produce a clean winner, and the winner will look like an answer. It also measures only the run you did: a heading-aware configuration validated on this quarter's Confluence export says nothing about the scanned vendor PDFs the same index will be carrying next quarter, because the strategy's assumption about the corpus is different for those documents.

And a retrieval metric, however carefully constructed, sits upstream of the thing anyone cares about. Span containment@10 says the answer text reached the context window. Whether the model then used it, and whether the cited chunk was the one that contained it, is a separate measurement with its own failure modes.

For chunking specifically there is one more limit worth stating before running any sweep at all. Two configurations can differ only in chunk size and produce a difference that has nothing to do with boundaries, because the larger size crossed a limit in the embedding model and the extra text never reached the vector. A size sweep that has not first established what the encoder actually accepts is measuring truncation and calling it segmentation.

That last failure has the same shape as the confounds above and one property that makes it harder to catch. Nothing in the run manifest records it, because nothing in the configuration changed except the variable you meant to change. The second cause was introduced by the first, so a hash over the pinned fields sees two runs that differ in exactly one place and passes them both.

A nine-point gap is not evidence about boundaries unless boundaries were the only thing that moved, and the discipline that makes it evidence is cheap to describe and expensive to run. One parsed corpus, cached under a hash. One query set, frozen before any result is seen, with answer spans marked. One embedding model, one index configuration, one value of k, one evaluator. A comparison function that hashes the pinned fields and refuses to print a number when they disagree, so the discipline survives the week somebody is in a hurry.

What that buys is a difference you can attribute, which is less than it sounds and more than most experiments deliver. It still says nothing about whether the configuration you picked is any good, nothing about the documents arriving next quarter, and nothing about what the generator does with the text once it arrives. And it is worth nothing at all if the metric underneath it cannot see the axis you moved — which is why span containment belongs beside recall from the first run rather than after the first argument about a result.