Embedding-based deduplication and near-duplicate detection

Read these first

• Similarity metrics and matching the metric to the model

What you will know

• Find near-duplicates in a corpus using embeddings

  A single source split into chunks can create false clusters, while boilerplate and shared templates can make unrelated pages look close.

• Choose a similarity threshold and defend it

  A cutoff that catches paraphrases may merge merely related documents, so calibrate it against labeled pairs and record false-merge and false-miss rates.

• Show duplicates crowding out diverse results in top-k

  Compare baseline and deduplicated results using unique-source counts or a diversity metric rather than visual inspection alone.

• Decide whether to drop, merge, or keep duplicates

  Dropping loses provenance, merging can erase conflicting updates, and keeping markers preserves auditability at the cost of ranking complexity.

• Explain the effect of a duplicated document on retrieval quality

  Repeated vectors increase the chance that one source occupies several slots, masking independent evidence and making apparent relevance scores misleading.

Chapters

1. 1

   A document appearing three times spends three top-k slots on one piece of evidence

   Retrieval returns ten results, and if three of them are the same content under different URLs, the context window holds eight distinct sources at best. Worse than the lost capacity is what repetition does downstream: a model reading the same claim three times treats it as corroborated, so a wrong answer arrives with more confidence than a correct one built from independent passages.

   Read chapter
2. 2

   An all-pairs comparison is quadratic, so the candidate set comes from a self-join

   Comparing every vector against every other is not available on a real corpus. Strip byte-identical copies with an exact hash pass first, since they cost nothing to find, then produce candidates with an ANN self-join: query each vector against the index for its top-m neighbours above a loose floor, or block on a cheap key first. The consequence to carry forward is that this search's own recall is a hard ceiling on every duplicate the pipeline can ever find — a genuine pair the self-join never surfaced is invisible to any threshold you choose later.

   Read chapter
3. 3

   Two clusters of high-similarity pairs are not duplicates at all

   Run that self-join and thousands of pairs above 0.97 come back — mostly adjacent chunks of one document, which must be excluded by source before any threshold is applied. The second trap is structural: two release-notes pages sharing a template score 0.96 on the boilerplate while differing in exactly the version numbers and fix lists a user is searching for.

   Read chapter
4. 4

   A threshold is calibrated against labelled pairs, never inherited from a blog post

   Cosine 0.95 means different things for different models, and spot checks confirm whatever you already believed. Sample candidate pairs stratified across the score range so most of your labels land near the boundary you are placing, label them duplicate or not, and read off the false-merge and false-miss rates at each cutoff. That table is the artefact you hand to someone who calls the number arbitrary — with the caveat stated on it that the false-miss rate is conditional on the candidate search, since pairs the self-join never returned were never available to be labelled.

   Read chapter
5. 5

   Recall and nDCG are blind to crowd-out by construction

   Both reward finding a relevant document, and a duplicate is relevant, so 0.88 recall@10 and 0.81 nDCG@10 stay healthy while users say the assistant keeps citing the same thing. Measure unique sources per top-k instead, comparing a baseline run against a deduplicated one. Raising k does not help, because the duplicates simply fill the larger window too.

   Read chapter
6. 6

   Dropping, merging and marking lose three different things

   Deleting the extra copies loses provenance, so you can no longer answer which system the answer came from. Merging into one canonical record can erase a genuine divergence where two copies were updated differently. Keeping every copy, tagging the duplicate cluster, and collapsing at retrieval time preserves auditability and costs you ranking complexity plus a collapse step on every query.

   Read chapter
7. 7

   Collapsing at query time silently shrinks k unless you over-fetch first

   The recommended policy is a retrieval-time operation, so it needs building: write the cluster id onto every chunk, fetch more than k so that collapsing a cluster still leaves k results to deliver, and decide which member represents the cluster — freshest revision, most authoritative source, or highest scorer, and the three disagree often enough to matter. The collapse also reshapes the ranking, because a source that occupied three slots now occupies one and everything below it moves up, and it adds a per-query cost you have to fit inside the latency budget. This is the run you compare against the baseline when counting unique sources per top-k.

   Read chapter
8. 8

   A 12% corpus cut that improves your metrics deserves suspicion first

   Improved retrieval metrics after removing an eighth of the index are consistent with a clean win and equally consistent with having deleted documents that were competing with your judged positives. Check what was actually removed by sampling merged clusters, confirm no judged relevant document was among the casualties, and look for per-query regressions on the long tail that the aggregate absorbed.

   Read chapter