A document appearing three times spends three top-k slots on one piece of evidence

A retriever hands a fixed number of results to whatever comes next, and that number is the whole of what the next stage gets to see. Top-k is the name of the budget: the retriever returns the k highest-scoring chunks for a query, and k is chosen against context length and cost rather than out of generosity, so it is 5, 10 or 20 far more often than it is 100. Scoring happens one chunk at a time against the query, which means nothing in the arrangement can notice that two of the k are the same content. A chunk that repeats another chunk is scored on its own merits and takes its own slot.

Two passages are near-duplicates when they are not identical byte for byte and carry the same content closely enough that a reader who has seen one of them learns nothing from the rest. Finding them across a collection of documents is near-duplicate detection, and detection together with the decision about what then happens to the copies is deduplication. You meet the problem on any corpus assembled from more than one system, which by now is most of them — a knowledge base wired to a wiki, a helpdesk and a shared drive at once will hold the same policy three times before anybody notices. It is also standard practice one layer down, in the pipelines that assemble training data from crawled text, where the same machinery is run for a different reason: copies inflate what a model memorises and waste the compute spent on them.

Ten slots, six pieces of evidence#

The budget is fixed before any query arrives. A prompt template concatenates the k retrieved chunks and the model sees whatever fits, so every slot spent on a copy of something already in the window is a slot that cannot carry an independent claim. That is the whole of the capacity argument, and it is worth watching it happen on one result set before generalising it.

One thing in that sentence is a design decision and the other is a requirement, and mixing them up is how the problem gets misdiagnosed. That k is small is a decision: somebody chose it against a context length and a per-query cost, and it could be raised tomorrow. That the k slots are all a downstream stage ever sees is a requirement, because a chunk not retrieved is not in the prompt and nothing later can consult it. The second fact is what makes the first one expensive, and it is also why the remedy has to act on which chunks are eligible rather than on how many are delivered.

Here is a result set from a support assistant at a payments company, answering a question about how long a customer has to dispute an invoice. The chunk text is abbreviated and the cluster column marks which chunks carry the same content:

Count the distinct contents rather than the rows. Cluster A holds three rows, clusters B and C hold two each, and three rows belong to no cluster at all. That is three groups plus three singletons, so six distinct pieces of evidence arrived in ten slots. Forty percent of the window carried nothing the model had not already read.

The four wasted slots can be arrived at a second way, and the second way is the one that generalises. Each cluster delivers one piece of evidence and occupies as many slots as it has members, so it wastes one slot fewer than its size: cluster A wastes two, and clusters B and C waste one each. Adding those gives four, the same four. Written generally, the waste in a window is the sum over its clusters of the size of each cluster minus one. A singleton contributes nothing to that sum, which is why the three uncoloured rows are irrelevant to the arithmetic.

That formulation says something the percentage does not. The waste depends only on how many copies of each thing are present, never on where in the ranking they landed. Three copies at ranks 1, 2 and 3 cost exactly what three copies scattered across ranks 1, 5 and 9 would cost, and the arithmetic does not change if the whole window is reordered. Anything that hopes to recover the loss by re-ranking is therefore working on the wrong quantity.

The size of the loss and the size of the rescue are the same number, and that is the sharpest thing in the table. Four slots were spent on copies, so buying them back promotes exactly four results from below the line, which are ranks 11 through 14. The billing handbook passage carrying the current sixty-day rule sat at rank 13, inside that span. Its recovery was not luck. It was guaranteed by the count of copies, because any passage within four ranks of the cut line is reachable the moment the four redundant slots are freed, and any passage further down is not.

That the numbers are typical rather than chosen can be checked independently. On the 4.2M-chunk index this assistant ran against, drawn from 610,000 documents across nine connectors, the mean number of distinct contents in a top-10 was 6.4. One worked query and a corpus-wide average landed on the same figure, which is what licenses reading the table as an ordinary result rather than as a specimen picked to make a point.

The cost is easy to underestimate because it does not look like a cost. Nothing is missing from the response, no error is logged, latency is unchanged, and the answer is fluent. The lost capacity is invisible precisely because the thing it would have contained never appeared.

Repetition reads as corroboration#

The capacity argument is the smaller half. The larger half is what happens when a language model reads the same claim three times in one context window.

Models are sensitive to how often a claim appears in their input, and the reason is worth stating as a chain rather than asserted. A model produces its answer conditioned on the whole prompt at once, so every passage in the window is evidence competing with every other passage. Nothing in a concatenated prompt records where each passage came from, or that two of them descend from one original, because the prompt is text and provenance is metadata that was dropped when the chunks were joined. With no origin information available, the only thing that distinguishes a claim made three times from a claim made once is that it was made three times. Agreement between passages and repetition of one passage therefore arrive in the same form, and there is no step anywhere in the model where the two could be separated.

If the prompt contains three paragraphs asserting a thirty-day window and one paragraph asserting sixty days, the thirty-day claim has the shape of consensus. The model does what a person skimming the same four passages would do, which is to treat the majority reading as settled and the minority reading as an outlier or an obsolete note.

That effect is measurable, and the measurement is built so that one term cancels. On a 200-question internal evaluation containing contradictions — pairs of passages where an older claim had been superseded by a newer one — the assistant chose the older claim on 81% of questions when the older claim appeared in three copies and the newer one appeared once. The same 200 questions, against a context in which each distinct claim appeared exactly once, put the older claim ahead 47% of the time.

Read those two runs as a subtraction rather than as two scores. Same questions, same model, same underlying difficulty in telling an old claim from a new one; the only thing that differs between them is how many times the older claim was written down. Whatever makes recency hard is present in both runs at the same strength, so it cancels, and the 47% is the size of that shared term. What is left after the cancellation is one-directional: 34 points of preference for the stale answer that the corpus manufactured rather than the model. The residual only runs one way because copying can add agreement and cannot subtract it.

The failure mode this produces is worse than a simple wrong answer, because the wrongness comes with corroboration attached. An answer built from three copies of one source and an answer built from three independent sources are indistinguishable in the response, in the citation list, and in most logging. A reviewer spot-checking the assistant sees three URLs, opens one, finds it says what the answer says, and moves on.

That is worth naming as a defence rather than as an inconvenience, because the citation list is the artefact users and reviewers actually trust. It reads identically in both of the states it is supposed to tell apart. Three URLs under a corroborated claim and three URLs under one paragraph pasted twice produce the same display, the same length, and the same impression of independent agreement. A check whose output does not vary with the thing it is meant to discriminate is not a weak check; it carries no information at all about that thing, and duplicates are the case it corrupts most directly.

Confidence signals inherit the same defect one layer up. Teams frequently build a heuristic on top of retrieval — flag the answer as low-confidence when the top results disagree, or when only one source supports the claim — and a duplicated corpus feeds that heuristic three agreeing sources for a claim that has one. The guardrail is then most confident exactly where the evidence base is thinnest, which is worse than having no guardrail, because the failure is now signed as safe.

Where the copies come from#

Duplicates are not a sign of a badly run knowledge base. They are the normal output of organisations doing reasonable things over several years:

The similarity column is where the problem stops being a database problem and becomes an embeddings problem, and the reason is in the shape of the column rather than in any single value. Only the first row is a point. Every row beneath it is a range, and the ranges overlap each other freely, so no horizontal line drawn through that column separates one origin from another. A hash reaches the first row and nothing below it.

An embedding is the vector a model assigns to a passage, positioned so that two texts saying the same thing land close together whatever words they use, which is the geometry the similarity metrics of topic 8.5 report on. The instrument has to be that one because the cheap one does not reach far enough. A hash finds copies that match byte for byte, and the copies that cause the damage never do. A PDF export differs from the page it came from in whitespace, in heading markers, in a page-number footer, and sometimes in one sentence that changed after the export. A pasted runbook differs in the two sentences somebody edited afterwards, and a migrated article differs in the markup its new system emits.

The consequence is a pair of documents that are the same evidence for every purpose a retriever cares about and share no exact hash at all. Their embeddings, on the other hand, sit at cosine 0.98 in a space where unrelated documents about the same product sit around 0.55, which is what makes embeddings the right instrument for finding them.

Chunk overlap, storage cost and the prompt-time filter#

Chunk overlap is the first thing this gets confused with. Most chunkers are configured with a deliberate stride overlap — 512-token chunks with 64 tokens of carry-over, so a sentence spanning a boundary appears in both chunks. Those chunks are near-identical at the seam and they are supposed to be. They come from one document, they are already attributable to one source, and collapsing them is a chunking decision rather than a deduplication one. Chapter 3 returns to this, because adjacent chunks are the single largest source of false positives in a naive detector, but the underlying defect is different: overlap is deliberate redundancy inside one source, and duplication is accidental redundancy across sources.

Storage cost is the second confusion, and it is the one that gets deduplication funded for the wrong reason. Trimming an eighth off a 4.2M-chunk index saves real money — roughly 3.1 GB of vectors at 768 float32 dimensions, plus the index structure — but that saving would not justify the engineering on its own, and framing the work as a storage project leads to the wrong design. A storage project deletes copies. A retrieval-quality project usually should not, for reasons chapter 6 works through in detail.

Deduplicating inside the prompt is the third, and it is tempting because it is a ten-line filter. Retrieve ten, compare the chunk texts, drop the ones that repeat, send what remains. That does prevent the model from reading the same paragraph three times, and it does nothing about the capacity loss, because the filter runs after the window has been chosen. A passage at rank 13 is not in the ten results the filter was handed, so no amount of comparing those ten to each other can promote it. The filter ships eight chunks instead of ten and calls it a fix. Handling duplicates properly means dealing with them at or before retrieval, so the slots they would have consumed get refilled from further down the ranking.

Finally, this is not a problem that better embeddings solve, and the reason is the one that makes the whole topic necessary. A stronger model draws copies of the same content closer together rather than further apart, because they are the same content. Improving the retriever improves the precision with which it returns the duplicate cluster.

The assistant above needed to know that ranks 1, 2 and 3 were one thing. That knowledge does not exist anywhere in the system at query time — the index holds 4.2M vectors and no notion of which of them are copies of which others — so it has to be computed, offline, over the whole corpus, before any of the retrieval-time machinery has anything to act on.

The obvious way to compute it is to compare every vector with every other vector and keep the pairs above some similarity. On 4.2M chunks that is 8.8 trillion comparisons, which is not a job you run; it is a job you cancel. The next chapter is about getting a candidate set of plausible duplicate pairs without doing that, and about the property of every method for doing so that constrains everything downstream: whatever the candidate search misses, no threshold, policy or evaluation later in the pipeline can ever recover.