Collapsing at query time silently shrinks k unless you over-fetch first

A duplicate group is knowledge sitting in a database until something acts on it, and the cheapest place to act is at query time. Every chunk carries the identifier of the group it belongs to, retrieval runs unchanged, and a collapse step then keeps one member of each group and discards the rest before the results are handed on. Nothing is deleted, every copy stays retrievable for whoever is entitled to it, and a wrong grouping decision is undone by reverting an identifier rather than by re-ingesting a source system.

The step has one property that decides how it must be built: it is subtractive. A collapse can shorten a list and can never lengthen one, so a caller who asked for k results and got a collapse applied to them receives fewer than k, and receives no error saying so. That is the whole engineering problem, and the fix has a name — over-fetching, retrieving more candidates than the target so that the survivors still number k — together with a second decision that the first one forces, which is which member of a group gets to be the survivor.

How much to over-fetch#

The over-fetch factor is not a constant to be guessed; it is a quantile of a distribution you already have from chapter 5's instrumentation. For each query in the log, record how many candidates had to be examined before ten distinct clusters had been seen, and read the answer off the tail. The last column of the table counts the candidates that were hydrated for nothing — hydrating a candidate means fetching its text and metadata from the primary store so it can be delivered, and every over-fetched result that the collapse then discards has been paid for.

The fourth column carries no information at all, and noticing that is the first step to reading the table. It is the first column minus the target, in every row: fetch fourteen for a target of ten and four candidates are surplus, fetch thirty and twenty are. It cannot be otherwise, because the target is fixed and every candidate beyond it is either a discarded copy or an unused spare. What that column contributes is not a measurement but a unit conversion, turning the fetch size into the cost of the fetch size.

The two columns that do carry information are the first two, and they have to be read as steps rather than as levels. Going from ten to fourteen removes about seventeen points of shortfall. Fourteen to sixteen removes twelve more. Sixteen to twenty removes nine. Twenty to twenty-four removes three. Twenty-four to thirty removes half of one. The return per extra candidate falls at every step and falls fast, which is what makes a permanently larger fetch a bad way to buy the last fraction of a percent.

The third column reaches its ceiling in the same row that the second column reaches its floor, and the ceiling is not a coincidence. Mean distinct delivered cannot exceed ten, because ten is what was asked for and the collapse discards the rest, so that column is bounded above by the target and saturates the moment nearly every query can fill its window. Two columns hitting their limits together is what identifies the operating point without any judgement being applied.

Fetching 24 for a target of 10 covers 99.4% of queries in one round trip, which is where the payments assistant settled. The remaining 0.6% are queries against genuinely pathological regions of the corpus — the ones whose top candidates are all copies of one export — and they take a second search with a larger k' and the already-seen cluster ids excluded. Handling them with a retry rather than by setting k' to 60 for everybody is the right trade, because the retry costs one extra round trip on six queries in a thousand while a permanently larger k' costs hydration on every query.

Two details in that retry are easy to get wrong. The second search has to exclude the clusters already collapsed rather than the chunk ids already seen, otherwise it returns the fifth copy of the same document. And the fallback has to be bounded — one retry, then deliver what you have — because a query whose entire neighbourhood is one cluster will otherwise walk the whole index looking for a tenth distinct source that does not exist.

A single fixed k' also assumes the redundancy profile is stable, and it moves whenever the corpus does. Adding a connector that mirrors an existing one shifts the distribution and the 99.4% coverage quietly becomes 94%. Recomputing the quantile weekly from the logs and alerting when the required k' moves by more than a few is a five-line job that catches a class of regression nothing else reports.

The shortfall this chapter opens on also argues for an invariant rather than a metric, and the difference between those two instruments is worth being exact about. A metric on the count of delivered passages produces a number that somebody has to look at, and a low number on that chart is indistinguishable from a quiet day with fewer queries or an unusual mix of them. Its reading is the same in the state where the collapse is starving the prompt and in the state where nothing is wrong. An assertion in the retriever that the collapsed list has exactly the requested length is evaluated on every query, fires on the event itself rather than on an aggregate of it, and emits a warning carrying the query and the cluster ids — which turns a silent degradation into a log line with everything needed to reproduce it. On the payments assistant that assertion fired 68 times in the first week, all of them on queries against one PDF archive whose entire contents were exports of one another, and the fix was a chunking change to that connector rather than anything in the retrieval path.

Which copy speaks for the cluster#

Collapsing means choosing, and there are three defensible rules that disagree with each other more often than intuition suggests. Measured across the 4,100 clusters on the payments corpus, all three agreed on the representative for 61% of clusters; highest-score and freshest agreed on 68%, and source-authority and freshest on 74%.

Those three figures are constrained by arithmetic and the constraint is what makes them readable. Any pair of rules must agree at least as often as all three do, since every cluster where all three agree is also a cluster where any two of them agree. So the three-way figure is a floor, and the distance from it to each pairwise figure is exactly the region where the third rule stands alone. Roughly one cluster in six has source-authority parting company with the other two, and rather more than that has highest-score doing so. Disagreement on a quarter to two fifths of clusters is not a rounding difference to be settled by preference.

Cluster 8841 from the previous chapter is a clean case of them disagreeing in the worst direction. The highest-scoring member for the query invoice dispute window was the MediaWiki copy from 2022, because migration had stripped its surrounding navigation and left a tight 180-token chunk that matched the query more purely than the 480-token Zendesk article containing the current 60-day rule. Selecting by score chooses the stale copy, and does so because it is stale — the newer page had grown a preamble.

The chain there is short and worth walking, because it looks like bad luck and is not. A similarity score is computed over a whole chunk, so a chunk that contains the query's subject and little else concentrates its signal, while a chunk that contains the same subject inside a longer body dilutes it. Abandonment strips a page of the surrounding material a live page accumulates. Being abandoned therefore tends to make a chunk shorter and more focused, which tends to make it score higher. Score is measuring concentration and is being read as authority, and on a migrated corpus those two point in opposite directions systematically rather than occasionally.

from datetime import datetime

AUTHORITY = {"zendesk": 3, "confluence": 2, "notion": 2, "drive": 1, "mediawiki": 0}

def choose(members, policy: str):
    if policy == "score":
        return max(members, key=lambda m: m.score)
    if policy == "fresh":
        return max(members, key=lambda m: (m.updated_at, m.score))
    if policy == "authority":                    # authority, then recency, then score
        return max(members, key=lambda m: (AUTHORITY.get(m.connector, 0),
                                           m.updated_at, m.score))
    raise ValueError(policy)

def collapse(hits, k: int, policy: str = "authority"):
    groups: dict[str, list] = {}
    for h in hits:                               # hits arrive score-descending
        groups.setdefault(h.cluster_id or f"single:{h.chunk_id}", []).append(h)
    out = []
    for cid, members in groups.items():
        rep = choose(members, policy)
        rep.cluster_size = len(members)
        rep.cluster_members = [m.source_url for m in members]
        out.append(rep)
    out.sort(key=lambda h: -max(m.score for m in groups[h.cluster_id]))
    return out[:k]

The three policies scored differently end to end on the 200-question evaluation: 0.79 for highest-score, 0.83 for freshest, and 0.86 for authority-then-recency. That last figure is the same one the final chapter reports for the mark-and-collapse configuration measured against three other runs, which is a useful check: two evaluations set up for different purposes landing on one number means the representative policy is carrying that difference rather than something else in the pipeline. The ordering is a fact about this corpus, where one connector really is the system of record for policy and the rest are copies of it, and it would invert on a corpus where the authoritative system is the slowest to be updated. The measurement is cheap enough to run for all three whenever the corpus composition changes.

Ordering the collapsed list is a decision hiding inside the collapse. Sorting the survivors by the representative's own score reorders the list whenever the representative was not the top-scoring member — cluster 8841's Zendesk article scored 0.03 below the MediaWiki copy, so ranking by the representative pushes the cluster down two places for no reason related to relevance. Ranking each cluster by its best member's score, as the code above does, keeps the cluster where the retriever put it while still delivering the copy you chose. Two questions are being kept apart there: where this evidence belongs in the ranking, which the retriever answered, and which copy of it the reader should be given, which the policy answers.

Two fields on the representative do the work chapter 6 was protecting. cluster_size lets the interface say that this content appears in four places, and cluster_members carries the URLs, so the audit question has an answer at the response level and the user sees the copy they can actually open. Without them, the collapse reintroduces the provenance loss that the marking policy exists to avoid, just at query time instead of at index time.

The ranking is not the same ranking#

Removing three chunks from a top-24 does not leave a shorter version of the same list; everything below the removals moves up, and the composition of the delivered window changes in ways downstream components notice.

The passage carrying the current sixty-day rule was at rank 13 in the baseline and arrives at rank 7. That is the whole point of the exercise, and it is also the moment several assumptions elsewhere stop holding.

A cross-encoder reranker sitting after retrieval — a model that reads the query and one candidate passage together and scores that pairing directly, rather than comparing two vectors that were computed without knowledge of each other — now receives a different candidate set, so its measured gain over the baseline retriever changes and has to be re-measured; a reranker tuned on duplicate-heavy candidates was partly earning its keep by pushing copies apart. Prompt templates that hardcode ten passages now receive ten genuinely distinct passages averaging different lengths, so token budgets shift — the payments assistant's median prompt grew from 5,100 to 6,350 tokens after collapse at the same k, because the copies it had been sending were the short export variants. That growth happened with k unchanged, which is the kind of thing a budget keyed to k will not catch. Any consumer that paginates has to carry the set of collapsed cluster ids into the next page request, or page 2 opens with the copies page 1 removed.

Paying for it inside the latency budget#

The added cost is small when the cluster id is available without a second lookup and large when it is not. Measured on the payments stack, against a p95 retrieval budget of 120 ms — p95 being the latency that 95% of queries come in under, and p50 the median — the accounting is:

Comparing the second and third columns row by row, only one row differs. The search costs the same, the collapse costs the same, and the metadata fetch goes from a batched 2.8 ms to 24 candidates at 1.1 ms apiece. The rows that do not move are what prove the case: whatever is expensive in the third column, it is not the over-fetch and it is not the collapse, because both of those are present in the second column at a price nobody would argue about.

The difference between the second and third columns is one design decision: whether cluster_id lives in the vector index payload, which is the small block of metadata a vector store keeps beside each vector and returns with every hit, or has to be looked up per candidate in the primary store. Storing it in the payload costs 16 bytes per vector — 67 MB across 4.2M chunks — and removes the entire third column. Any system where collapse is planned should write the cluster id into the payload at index time even before the collapse is built, because retrofitting it later means a full index rewrite.

The 3.1 ms the ANN search costs to go from k=10 to k'=24 is the difference between the first two entries of the top row, and it is the search's own scaling with k rather than anything the collapse introduced, which makes it unavoidable. It is also small, because HNSW's work is dominated by graph traversal rather than by the size of the result heap: more than doubling the number of results asked for adds well under half again to the search time.

Three things collapse leaves exactly as it found them#

It does not rank. Collapse only removes candidates, so a retriever that ranks badly ranks badly over a shorter list, and teams sometimes reach for deduplication when the actual complaint is that the right document is at rank 40.

It does not touch redundancy below the threshold. The 12,800 genuine duplicate pairs living in the 0.88 to 0.92 band that chapter 4 showed to be economically unreachable are still in the index, still occupying slots, and still invisible to this machinery. Collapse enforces the decision the calibration made; it does not improve it.

And it does not control what the generation step does with the cluster. A response that cites all four members of cluster A, taken from cluster_members, has restored the appearance of corroboration in the user interface even though the model only read one passage. That is the citation-list problem of chapter 1 arriving through the fix rather than through the fault, and it is worth noticing that the collapse's own provenance field is what makes it possible. Rendering a collapsed cluster as one citation with a "also appears in 3 other places" affordance keeps the provenance available without recreating the impression of independent agreement.

There are now two systems that can answer the same query set: the baseline, and this one. Running both across the 1,200-query evaluation and the production log replay produces the comparison chapter 5 specified — unique clusters per top-k, cluster-recall, answer accuracy — with the threshold, the cluster run id, the representative policy and the over-fetch factor all recorded next to the numbers.

That comparison is where deduplication projects are usually declared finished, and it is the point at which the results deserve the most scepticism. Removing an eighth of a corpus and watching retrieval metrics rise is consistent with having fixed the problem and equally consistent with having deleted documents that were competing with the very documents your evaluation calls correct. Telling those apart is the last chapter.