An unchanged average may mean the merge rule never fired

The expansion strategies of the previous two chapters transform every hit the same way. Parent-document retrieval dereferences each matched child to its enclosing section; sentence-window retrieval pads each matched sentence with its neighbours. Auto-merging is the first strategy in this topic that looks at the shape of the whole result list before deciding whether to act. The rule is the standard one: when three or more sibling chunks from the same parent appear in the top-k, drop them and insert the parent in their place. Everything below follows from the word when.

A rule that fires on some queries and not others cannot be evaluated the way an unconditional one can, and the arithmetic that shows why is short enough to do in your head. On the queries where the condition does not hold, the two systems being compared build the same prompt from the same passages and produce the same answer, so they contribute an identical figure to both sides of the comparison. Whatever the strategy is worth shows up only on the remainder. If the remainder is small, an aggregate over the whole set is mostly a measurement of agreement between a configuration and itself, and it will read as "no effect" whether the strategy is excellent, useless or harmful on the queries it actually reaches.

The arithmetic that makes an aggregate uninformative#

The relationship is worth writing down, because it is the reason conditional retrieval strategies are so often mis-evaluated:

Δ_overall = fire_rate × Δ_on_fired_queries

Read it as a statement about where the movement can come from. The unfired queries contribute zero to the difference by construction, since both systems produced the same answer on them. The fired queries contribute their own difference, and that difference is diluted by the share of the set they represent. At a fire rate of 4%, a strategy that improves the queries it touches by a full 12 points moves the headline number by 0.48 points, which is 4% of 12 and nothing else.

Set that against what the measurement can resolve. On 400 queries, the 95% confidence interval on an accuracy estimate around 73% — the band the true value plausibly falls in, given that 400 queries are a sample rather than the whole of your traffic — is roughly ±4.4 points. A genuine 12-point effect on the affected subset therefore arrives at the headline an order of magnitude inside the noise floor. The experiment could not have detected the thing it was run to detect, whatever the result had been, and that is knowable before the experiment runs from two numbers: the fire rate and the size of the set.

The fix is to report the stratified numbers rather than the pooled one. Split the eval set by whether the condition fired, and compare within the fired stratum only — where the two systems actually differ.

The first two rows are counts wearing percentage signs, and converting them back is the quickest way to see how little is being measured. Sixteen queries at 56.3% is nine correct, and sixteen at 68.8% is eleven; the fired stratum is 9 of 16 against 11 of 16, a difference of two queries. Three hundred and eighty-four at 74.2% is two hundred and eighty-five correct on both sides.

Those counts reconstruct the flat-chunk pooled figure exactly. Nine correct in the fired stratum plus two hundred and eighty-five in the unfired one is two hundred and ninety-four of 400, which is 73.5% — the number in the bottom-left cell, with no residual. That closure is worth having because it establishes that the strata and the pooled row are describing the same 400 queries rather than two different runs, and it licenses using the same route on the other column.

The same route on the auto-merged column does not land on the tabulated cell. Eleven correct plus two hundred and eighty-five is two hundred and ninety-six of 400, which is a shade under seventy-four per cent rather than 73.4%, and the identity at the top of this section gives the same answer from the other direction: 4% of a 12.5-point gain is a rise of half a point, not a fall of a tenth. Two independent routes — the counts, and the fire-rate identity — agree with each other and disagree with the tabulated pooled cell, while the prose count of 9 of 16 versus 11 of 16 supports the strata. The pooled auto-merged figure is the one to distrust, and the movement it should show is a small rise.

Nothing in the chapter's argument turns on which of those it is, which is the substantive point rather than a way of excusing the discrepancy. Half a point up and a tenth of a point down are both far inside ±4.4. The measurement is uninformative at either value, and a reader who resolves the disagreement in favour of the more flattering number has still learned nothing about whether merging works here. That is what an uninformative aggregate looks like from the inside: it is not that it reports the wrong sign, it is that its sign is not carrying information.

Two things fall out of the stratified table. The strategy is doing what it claims on the queries it reaches, and sixteen queries is far too few to say so with confidence — 9 of 16 versus 11 of 16 correct has a confidence interval wide enough to include zero and to include +30. The honest conclusion after this run is that the measurement is not finished, and the next step is to raise the fire rate or to build an eval slice on which the condition fires often, rather than to close the ticket either way.

Notice also that the fired stratum has a much lower baseline, 56.3% against 74.2%. That is expected and it is informative: the queries whose evidence is scattered across several sibling chunks are the hard ones, which is why they are the queries a merge rule was built for. It also means you cannot compare the fired stratum to the overall baseline and call the difference an effect — the eighteen-point gap between those two baselines is a property of which queries fired, present before either system ran.

Instrumenting the condition#

Auto-merging operates on the rank list rather than on individual hits, which is the structural difference from the expansion strategies in the previous two chapters. Parent-document and sentence-window transform each hit independently. Auto-merging looks at the whole top-k, counts how the hits distribute across the hierarchy, and rewrites the list only where a parent has accumulated enough matched children. The hierarchy itself comes from parsing each document at several granularities at index time — commonly 2,048, 512 and 128 tokens, with every leaf carrying its chain of ancestor IDs — and only the leaves are embedded.

from collections import defaultdict

def auto_merge(hits, docstore, ratio_threshold=0.5, min_siblings=3, log=None):
    """Replace matched siblings with their parent when enough of the
    parent's children are present. Returns the rewritten hit list."""
    by_parent = defaultdict(list)
    for h in hits:
        by_parent[h.metadata["parent_id"]].append(h)

    merged, fired = [], []
    for parent_id, siblings in by_parent.items():
        n_children = docstore.child_count(parent_id)   # total, not matched
        ratio = len(siblings) / n_children
        if len(siblings) >= min_siblings and ratio >= ratio_threshold:
            best = max(s.score for s in siblings)
            merged.append(Hit(id=parent_id, text=docstore.get(parent_id),
                              score=best, n_tokens=docstore.tokens(parent_id)))
            fired.append({"parent_id": parent_id, "matched": len(siblings),
                          "of": n_children, "ratio": round(ratio, 2)})
        else:
            merged.extend(siblings)

    if log is not None:
        log.append({"fired": bool(fired), "merges": fired,
                    "tokens_before": sum(h.n_tokens for h in hits),
                    "tokens_after": sum(h.n_tokens for h in merged)})
    return sorted(merged, key=lambda h: h.score, reverse=True)

The log argument is the part worth insisting on. A conditional retrieval component that does not record whether it acted cannot be evaluated, because the fire rate is not recoverable afterwards from the answers. Adding the instrumentation after a disappointing result costs an extra evaluation cycle. Recording the fire rate, the matched-to-total ratio at the moment of firing, and the token delta gives you everything the rest of this chapter needs.

The n_children lookup is the other detail, and it is a small instance of the mistake this whole chapter is about. Counting matched siblings alone — "three or more from the same parent" — treats a parent with four children and a parent with forty identically. The guard is a count, and the quantity that decides whether merging is proportionate is a share. Three matched children out of forty triggers a merge that replaces 600 tokens of precise material with a 2,048-token section for which the system has evidence covering a fifteenth, and the guard reads exactly three in that case and in the case where three of four children matched. A threshold that cannot move when the danger moves is not a threshold, which is what the ratio guard is there to supply.

The two directions a threshold fails in#

Sweeping the ratio threshold on the same corpus makes the trade explicit. These runs hold min_siblings=3 and k=12 fixed and vary only the fraction of a parent's children required to fire.

The 0.50 row is the configuration the stratified table above measured, and the two agree where they touch: a fire rate of 4% and an accuracy on fired queries of 68.8% appear in both, produced by a threshold sweep on one side and by a stratified replay on the other. That agreement is what lets the rest of the sweep be read as the same experiment at other settings.

At 0.20 the rule fires constantly and the fired stratum gets worse than flat chunks, because most of those merges are pulling in a 2,048-token parent on the strength of two or three loosely related children. The audit column counts, over forty sampled merges, how many produced a parent where a majority of the text was unrelated to the query — the same parent relevance problem from chapter 2, now arriving as a consequence of a threshold rather than as a property of the documents. At 0.65 the rule effectively stops existing, and the queries whose evidence genuinely straddles a chunk boundary get no help at all; the fired stratum has too few members to score.

The token column is worth reading as gaps from the bottom row rather than as levels. At a 1% fire rate the rule is doing nothing on ninety-nine queries in a hundred, so 3,050 is very close to the prompt size of the unmerged pipeline, and every other row's excess over it is what merging spends. Those excesses are a couple of hundred tokens at a 4% fire rate, a little over two and a half thousand at 18%, and a little under five thousand at 31%. Divide each by its fire rate and the cost per fired query is not constant: it is a few thousand tokens at the top of the range and more than twice that at the bottom. Lowering the threshold does two things at once — it admits more queries, and it admits more merges within each query, since more parents clear the bar — so the per-query cost grows as the rate does. That is why the token column climbs faster than the fire rate, and it is a reason to treat a low threshold as more expensive than a linear reading of these rows suggests.

The middle rows are where the tuning happens, and the best fire rate and the best accuracy-on-fired do not coincide with the best pooled result. This is why the pooled number is a bad optimisation target for a conditional strategy: it will always prefer the threshold that does the least, because doing nothing has zero variance and the pooled figure is dominated by queries the rule never touched.

The failure that is harder to see is the second one — evidence split across a boundary where the rule does not fire, because only two children matched. Those queries look like ordinary retrieval failures in the logs, and the diagnostic is to take the queries the merged system got wrong, check whether the judged-relevant evidence spans two or more leaves of the same parent, and count how many of those had fewer matched siblings than min_siblings. On the corpus above, that was 23 of 106 wrong answers. That denominator is worth pausing on: a system scoring around 73.5% on 400 queries gets 106 of them wrong, so 106 is the whole failure population rather than a sample of it, and 23 is more than a fifth of everything the system got wrong. It is also a much larger population than the 16 queries the rule was firing on, which argues for lowering min_siblings to 2 while raising the ratio guard rather than loosening both.

Getting the fire rate into a range you can measure#

Four percent is not a law of nature; it is a consequence of parameters chosen without reference to it. Three of them move it directly.

Leaf size relative to parent size sets the ceiling, and the arithmetic is one division. With 512-token leaves under 2,048-token parents there are four children per parent, so a ratio threshold of 0.5 needs two of four — easy to satisfy inside a top-12. With 128-token leaves under the same parents there are sixteen children, 0.5 needs eight of them, and eight leaves of one parent cannot all appear in a list of twelve results unless two thirds of the list belongs to that parent. Checking that the threshold is satisfiable given the branching factor and k takes one line of arithmetic and catches the most common misconfiguration, where a rule cannot fire at all and the logs show a fire rate of exactly zero.

The retrieval depth before merging is the second knob, and it is nearly free. Merging operates on the rank list, so retrieving 30 candidates, merging, and then truncating to the prompt budget gives the rule three times the opportunity to see siblings together while sending the same amount of text to the generator. On the corpus above, moving pre-merge k from 12 to 30 took the fire rate from 4% to 19% at an unchanged ratio threshold, and cost 90 ms of extra retrieval and reranking. That is the change to make first, because it raises the fire rate without loosening the condition that keeps merges proportionate — the rule is unchanged and only its opportunities have grown.

The third is min_siblings, and lowering it from 3 to 2 while raising the ratio guard is usually a better trade than the reverse. Two of four children is genuine evidence that a section is the locus of the answer; three of sixteen is not, and the pair of parameters exists precisely so that the count and the proportion can disagree.

Even at a 19% fire rate, an eval set of 400 queries gives 76 fired queries, which resolves differences of about eight points. If the effect you are chasing is smaller than that, the general-purpose set is the wrong instrument and a targeted slice is the right one. Building it is mechanical when the judgments record which passages are relevant: select the queries whose judged-relevant passages include two or more leaves sharing a parent, which is the population the rule exists to serve, and label another eighty of them by hand if the harvest is thin. A 150-query slice on which the condition fires 90% of the time measures merging directly, and the pooled set then serves its proper purpose of confirming that nothing else regressed.

Production fire rate is worth logging separately from eval fire rate, because they diverge. Real traffic has a different query-length distribution and a much heavier head of repeated questions, and on the same system the production fire rate was 11% against the eval set's 4% — the eval set had been built to cover the corpus evenly, so it over-represented queries about rarely-touched documents where no two siblings ever rank together.

Boundary quality is decided before merging runs#

Auto-merging is not a repair for chunk boundaries that fall in the wrong place. If the leaf splitter cuts a table row in half, merging three siblings restores the table only when all three happen to be retrieved, and the query that matched one mangled leaf and nothing else is unaffected. Boundary quality is decided at index time by the chunking decisions from topic 9.5, and a merge rule operating on the rank list has no way to know that a leaf is malformed — the only thing it reads is which parent each hit belongs to.

It is also not the same lever as parent-document retrieval, despite both returning parents. Parent-document expands on evidence from a single hit and does so every time, so its cost is predictable and its risk is per-document. Auto-merging expands on evidence from the shape of the result set and does so rarely, so its cost is low in aggregate and its risk is concentrated in the queries where it fires. Running both at once is a common configuration and it needs care, because parent-document expansion changes what the top-k contains and can therefore change how often the merge condition is satisfiable.

And it presumes that a parent is the right container for the merged evidence. Merging assumes the answer's pieces are siblings — near each other in one document's hierarchy — which is a fair assumption for a policy manual and a poor one when the evidence for a question is spread over eleven documents.

An unchanged aggregate, then, is the beginning of an investigation rather than the end of one. Before it can mean that a conditional strategy failed, it has to be paired with a fire rate, and a fire rate low enough to make the aggregate insensitive is the ordinary case rather than the exception. Report the fired and unfired strata separately, check that the strata reconstruct the pooled figure, and treat the pooled figure as a regression guard rather than as the thing being measured.

The strategies so far have all treated the same failure, that a retrieved unit is too small a piece of the truth. The opposite failure exists and is at least as common in production: a retriever that returns five units, all sufficient, all saying the same thing, while the parts of the answer nobody wrote about twice never appear at all.