A shortened revision leaves its old chunks behind, unreferenced and fully retrievable

Updating a document in a vector index is a reconciliation between two sets of ids: the ids the new revision of that document produces, and the ids the index is already holding for it. A chunk is one piece of a document — a few hundred tokens of its text — embedded into a vector and stored under an id. An upsert is a write that inserts an id the index has never seen and overwrites whatever is stored under an id it has. An orphan is a chunk that survived an update it should not have survived: the index holds it, the current revision of its document no longer produces it, and the code that wrote the update does not know it is there.

An orphan matters because it stays retrievable. Every vector query returns a top-k — the fixed-size list of the k nearest stored chunks a query may return, k being whatever number the retrieval layer asks for — and an orphan competes for those slots while holding text somebody deliberately deleted. Orphans are made by a pipeline of the most ordinary kind: an ingest job that splits a document into chunks, embeds each one and writes the results to the index. Almost every such job updates a document by computing the chunks the current revision produces, upserting them, and reporting success — correct whenever the new revision produces at least as many chunks as the old one, which is most of the time, and wrong in exactly one direction.

The writer's set and the index's set are different sets#

The failure is not a bug in any particular line of code. It follows from a modelling assumption that is almost universal and almost never stated: that writing a document means writing the chunks the document currently produces. What an index actually holds for a document is whatever the last several writes left there, and only an operation that reads the index's current set can know what that is.

Work one shrinking edit all the way through, on a document stored under the id policy_travel. Its revision 4 produced fourteen chunks, so the index held fourteen points for it, numbered from zero: policy_travel:0 through policy_travel:13. The edit shortens the document and revision 5 produces nine. The writer computes those nine, upserts them under policy_travel:0 through policy_travel:8, and the index acknowledges nine successful writes.

An upsert acts on exactly the ids it is handed and on nothing else. For each of the nine ids in the call, the index either inserts a new point or overwrites the one already stored there; for every id not in the call it does nothing, and doing nothing to unnamed keys is what makes an upsert cheap. Nothing in that call describes the document as a whole, and nothing in the index connects policy_travel:13 to the write that just landed on policy_travel:0.

So the ids that survive untouched are the ids revision 4 produced and revision 5 does not. Revision 4 produced fourteen and revision 5 produced nine, and fourteen minus nine is five, so five points are left holding text from a revision that exists nowhere else in the system. Which five follows from the numbering rather than from anybody's choice: ordinals run from zero, revision 5 occupies zero through eight, and the ids above eight are nine, ten, eleven, twelve and thirteen. That is where policy_travel:9 through policy_travel:13 comes from.

Stated generally, an upsert of the new set over the old one is a partial function over the old set: it covers the intersection — every id both revisions produce — and leaves everything outside that intersection as it was. When the new set is a superset of the old, nothing is left outside and the update is correct by accident. When the new set is smaller, what is left outside is the difference between the two sets, and that difference stays in the index and stays searchable.

No error is raised, because no operation failed. An upsert of nine ids is a complete success even when the document previously occupied fourteen, so a writer checking return codes sees nine acknowledgements and reports a clean reindex. No correct answer to "did my nine writes land" is also an answer to "what else is in there".

Where fourteen and nine come from is worth one more step, because it lets you predict an orphan count without opening the index. A chunker with a fixed size and a fixed overlap advances through a document by a stride equal to its size minus its overlap: it emits a window of text, then begins the next window that far along. The overlap is the text appearing in two consecutive chunks; the stride is the new text each chunk adds. The number of chunks a document yields is therefore roughly its length divided by that stride.

The part to rely on is the proportion rather than the absolute count. Stride is a constant of the chunker, fixed by its configuration and identical for every document it processes, while length is a property of the document. Dividing by a constant preserves proportion, so a document losing a given fraction of its length loses about the same fraction of its chunks. The figures here behave that way: the edit took the policy from about 9,400 characters to 6,100, a bit more than a third gone, and the chunk count went from fourteen to nine, also a bit more than a third.

Two cautions belong on that division. A length quoted in characters and a chunker configured in tokens are not directly divisible, and nothing above asks you to divide them — the proportion holds under either unit, because a constant stride divides out of a ratio however it is measured. And it licenses an estimate rather than a count: apply the fractional drop in length to the chunk count the document had, and the result is roughly the size of the difference set. Boundaries land where the text lets them, so losing a paragraph from the middle is not the same as being truncated at the end. Even so, a length history alone tells you whether a body of documents has this problem, without touching the index.

Which chunks get left behind depends on how you mint chunk ids, and the three common schemes fail differently enough to be worth putting side by side.

Content-hash ids reward a walk, because their failure is the one that surprises people. The id of a chunk is a hash of its text, so an id changes exactly when its text changes. Boundaries, though, are computed by walking the document's token stream from the beginning and cutting every stride. So inserting one sentence into the introduction of a document shifts every cut after it by the length of that sentence, and every downstream chunk then holds a different span, hashes to a different id, and is written as a new point, while the old point under the old hash is never named and never removed. A one-sentence edit near the top of a forty-page document orphans the remaining thirty-nine, and the writer reports success, because every id it was asked to write did land.

The scheme feels safer than ordinals, and the reasons it feels safer are real. Hashing the text deduplicates: two identical chunks in different documents collapse to one point, so a boilerplate header repeated across a thousand documents is stored once. It also makes re-ingest idempotent, since replaying the same document produces the same ids and a retried job overwrites the same points rather than adding a copy. Both properties are genuine, and neither has anything to do with removing what the new revision no longer contains.

Ordinal ids are the most common scheme and, over a year, the most damaging, for a reason unrelated to how badly they fail on any one edit. They are correct for the majority case — a document edited in place at roughly the same length — and wrong only when it shrinks. A failure firing on a minority of edits and leaving a handful of points behind each time announces itself in no graph anybody keeps: an index is expected to grow, and this growth is smooth enough to read as content being added. Fresh UUIDs are the least harmful in practice for the opposite reason, since orphaning every previous chunk on every write makes storage climb at a rate somebody notices in the first week.

Three per cent of the index, and why that is not a storage number#

Put the same mechanism on a corpus — the whole set of documents a retrieval system has embedded and stored. Take one of 2.4 million chunks across 180,000 documents, with roughly 4% of documents edited per month and a mean of 2.1 orphans per shrinking edit. The figure usually quoted for that shape is about 3% of the index unreferenced after a year.

Walking the arithmetic exposes a parameter the sentence does not mention. Four per cent of a hundred and eighty thousand documents is a bit over seven thousand edits a month, and twelve months of that is on the order of eighty-six thousand edits. At a mean of 2.1 orphans each, if every one of those edits shrank its document, the year would end with something close to a hundred and eighty thousand orphans. Three per cent of 2.4 million chunks is about seventy-two thousand. The two do not meet, and the gap is a factor nobody stated.

That factor is the share of edits which shrink a document. Seventy-two thousand over a hundred and eighty thousand is about two in five, so the quoted figure assumes roughly two edits in five reduce a document's length, the other three lengthening it or leaving it about the same and producing no orphans under ordinal ids. This is not a criticism of the number, which is reasonable for documentation under active maintenance. It is what a reader has to know in order to transfer it.

For your own corpus the arithmetic is the edit rate, times the shrink rate, times the mean orphans per shrinking edit, divided by the size of the index. The third factor is the only one most teams have ever measured, usually because it fell out of a repair job. Edit rate comes free from any content store's timestamps. Shrink rate needs the length history the proportion above already asked for, and it decides whether a corpus lands anywhere near the quoted figure — a wiki whose pages only ever grow accumulates almost no orphans, and a corpus of regularly pruned policy documents accumulates far more than 3%.

None of that would be worth measuring if the leftover points were inert. An orphan is a previous revision of a document that is still in the corpus and still about its subject, so its text is largely the text of the live chunks around it. It is by construction a near-duplicate of live content. Text like that embeds to a nearby vector, and a nearby vector scores well against the same queries the live chunk scores well against. An orphan does not sit unreachable at the far edge of the space; it lands high in the ranking for precisely the queries its document was written to answer.

That is the step turning a storage statistic into a retrieval one. A top-k is a fixed-size list: ask for ten candidates and ten come back, whatever else is in the index. Every orphan outscoring a live chunk takes that chunk's place in the answer, so the cost is the slot rather than the disk. Three per cent scattered across the bottom of every ranking would be harmless, and three per cent that is by construction near the top is not the same three per cent.

A tombstone gauge and a recall check both read healthy#

This survives in well-run systems because the two instruments a careful team already has are both constant across the thing they would have to discriminate. Neither is broken, and each is a correct measurement of something adjacent, which is what makes it convincing.

A tombstone gauge counts vectors deleted and not yet reclaimed — a tombstone being the marker an index writes in place of a point when a delete commits, before compaction removes it. It is the standard signal for an index carrying dead weight, and what it is a statement about is reclamation: how far behind compaction is running. An orphan was never deleted. No delete was issued for it, so no tombstone exists and it contributes nothing to the count. The gauge reads the same on an index with no orphans and on one that is three per cent orphans, and it goes on reading the same as that number grows.

Recall measured against exact search is blind for a subtler reason, and it is the more convincing of the two, because it is the instrument a team reaches for once it suspects the index. It takes a set of queries, runs each through the approximate nearest-neighbour index, runs the same query as brute-force search over every stored vector, and reports the overlap. Both sides read the same stored vectors. The orphan is one of those vectors, so brute force retrieves it and calls it a true nearest neighbour, the approximate index retrieves it too, and the two agree about it perfectly.

What that measurement answers is "is my index structure returning what my corpus contains", and it answers it correctly. The corpus is what is wrong. A recall figure of a hundred per cent is compatible with an index in which not one document is at its current revision, because a perfectly faithful search over stale content is still a perfectly faithful search. Both instruments fail in the same shape, worth recognising in any defence: a measurement reading identically whether or not the condition it exists to detect is present detects nothing, however well calibrated it is.

A third instrument is worth naming because teams lean on it hardest, which is users noticing. Three months can pass before anyone raises this, and that gap is a property of the failure rather than of the team. A stale chunk produces an answer that is fluent, on-topic, and attached to a citation that resolves — the document exists, the link opens, the page is the right one. Only a reader who knew the removed text had been removed could tell the answer was wrong, and the people who know a clause was deleted are the people who deleted it, who are not the people asking about it months later. Retrieval failures that return nothing get reported within hours; retrieval failures that return a confident previous revision get believed.

What catches orphans is a comparison against the system of record, and there are two places to do it. The cheap one is an audit that walks documents and diffs chunk counts:

SELECT d.id,
       count(c.id) AS rows_in_db,
       v.chunk_count AS in_index,
       v.chunk_count - count(c.id) AS orphans
FROM documents d
JOIN document_chunks c ON c.document_id = d.id
JOIN LATERAL vector_chunk_count(d.id) AS v ON true
WHERE d.updated_at > now() - interval '90 days'
GROUP BY d.id, v.chunk_count
HAVING v.chunk_count <> count(c.id)
ORDER BY orphans DESC
LIMIT 100;

Restricting to recently updated documents keeps the audit affordable, since a document that has not been edited cannot have acquired new orphans, and it turns a full-corpus scan into a scan of the few per cent that changed. Run daily, this finds orphans within a day of their creation and gives you the list to repair. The other place is the query path, and it protects users while the audit is still catching up.

Reconciliation, and a revision key that outlives the writer#

An update is a reconciliation between two sets: the ids the new revision produces and the ids the index currently holds for that document. Written explicitly, the sequence is read the index's set, upsert the new set, delete the difference. The first of the three is the step an ordinary writer omits, and it is the only one that can see an orphan, because the difference set cannot be computed from the incoming chunks alone.

def reindex_document(doc_id: str, revision: int, chunks: list[Chunk]) -> Reconciliation:
    existing: set[str] = vectors.list_ids(filter={"doc_id": doc_id})
    incoming: dict[str, Chunk] = {f"{doc_id}:{c.ordinal}": c for c in chunks}

    vectors.upsert([
        Point(id=cid, vector=c.embedding,
              payload={"doc_id": doc_id, "rev": revision, "ordinal": c.ordinal})
        for cid, c in incoming.items()
    ])

    stale = existing - incoming.keys()
    if stale:
        vectors.delete(ids=sorted(stale))

    metrics.gauge("reindex.orphans_deleted", len(stale), tags={"doc": doc_id})
    return Reconciliation(written=len(incoming), deleted=len(stale))

The line stale = existing - incoming.keys() is the difference set from the worked example, computed rather than assumed: on that document it returns the five ids from nine through thirteen, and on a document that grew it returns empty and the delete never runs.

The upsert happens before the delete, and the ordering is an argument about how each version degrades rather than a matter of taste. The tempting simplification is to delete everything belonging to the document and then insert the new set. It is easier to write and it is correct in the sense that no orphan survives it. It also leaves the document partially unsearchable for however long the two operations take — typically 40 to 400 milliseconds, occasionally much longer if the delete triggers a segment merge and the engine rewrites a segment to reclaim space. A query about that document arriving inside that window matches nothing at all.

Upsert-then-delete has a window of its own, the same length. What differs is what sits inside it: the new chunks are already searchable and the stale ones have not gone, so a query landing in the window can retrieve one chunk of the previous revision alongside the current ones. Upsert-then-delete degrades to serving one stale chunk. Delete-then-upsert degrades to serving none.

Which is worse depends on who consumes the output, and for a retrieval system the consumer is usually an assistant that answers from whatever context it is handed. Given a slightly stale passage it produces an answer that is mostly right and attributable to a document the reader can open. Given nothing it answers from its own weights, which is the failure with no citation, no traceability and no upper bound on how wrong it can be. That window is entered every time a document is edited, thousands of times a month on a corpus of any size, and the version degrading to slightly wrong is the one to take the bet on.

Engines that support delete-by-filter let you replace the read-and-diff with a single conditional delete, which is both cheaper and more robust because it does not depend on your list of existing ids being complete:

vectors.upsert(points)                      # revision N
vectors.delete(filter={
    "must":     [{"key": "doc_id", "match": {"value": doc_id}}],
    "must_not": [{"key": "rev",    "match": {"value": revision}}],
})

That form deletes anything attributed to the document that does not carry the current revision, which covers orphans from ordinal ids, from hash ids, and from a half-finished write that crashed after four of nine upserts. It requires that the payload field is indexed for filtering, and it requires the revision to be monotonic per document, which is satisfied by an integer counter on the document row or by the updated_at timestamp if the CMS guarantees it moves forward.

The revision key earns its place a second time at query time, independently of any repair job. When retrieval returns a candidate, the payload says which revision the chunk belongs to, and a join back to the primary store says which revision is current. A candidate whose revision is behind is dropped from the candidate set before ranking and counted:

current = db.current_revisions([c.payload["doc_id"] for c in candidates])
fresh = [c for c in candidates if c.payload["rev"] == current[c.payload["doc_id"]]]
metrics.increment("retrieval.stale_candidates", len(candidates) - len(fresh))

That counter is the single most useful number in this chapter, because it measures the harm directly instead of standing in for it. Every other signal here is a proxy: a chunk-count diff says an orphan exists somewhere, a tombstone gauge says something was deleted, a recall figure says the index structure is intact. What retrieval.stale_candidates counts is candidates that reached the ranking stage carrying a revision behind the current one — occasions on which a user came within one ranking position of being shown deleted text.

It costs one indexed lookup per query, against a store you are already querying for document metadata. What it buys is a guarantee that is end-to-end. Every other defence here depends on a writer being correct: the reconciling writer, the delete-by-filter form, the audit that trusts the chunk table. The query-time filter depends on none of them, so whatever put a stale chunk in the index and whatever failed to take it out, the chunk is dropped before ranking and the drop is counted. A healthy system holds the counter near zero and spikes when a writer regresses.

Repairing the orphans that already exist is a separate job from preventing new ones, and worth sizing before scheduling. A sweep lists the index ids for each document and diffs them against the chunk table, running at roughly 300 documents per second against a warm store with a filter index on doc_id. Dividing 180,000 documents by 300 a second gives six hundred seconds, which is ten minutes of wall clock. No part of it is re-embedding, because repair only deletes and there is no model call in the loop, which is why a full sweep is priced in minutes where a reindex of the same corpus is priced in hours.

Restricting the sweep to documents whose updated_at has moved since the writer was fixed cuts it to a few thousand documents and under a minute, and that narrow sweep is a different instrument rather than a cheaper one. It can only find orphans hanging off documents edited since the fix, and every orphan created before the fix sits under a document whose updated_at may not move again for years. Running it hourly and never running the wide one leaves the accumulated backlog permanently invisible. Most teams therefore run the narrow sweep hourly as a safety net and the full sweep once, after deploying the reconciling writer.

Copies outside the index, and a primary store that is wrong#

Reconciliation keeps the vector index and the primary store in agreement on one question: which chunks belong to a document. It is not a substitute for the delete enumeration in chapter 1, where a delete is traced through every copy of a document a system holds. It says nothing about a result cache holding an answer built from the old chunk, an embedding cache still keyed by the removed text, or a summary generated from revision 4 and stored as a document in its own right. An edit that removes a sentence has the same fan-in problem as a delete, with the added difficulty that no one thinks of an edit as a deletion event at all — the ticket says update the policy, the audit log records an update, and whatever cache invalidation hooks anybody wrote are attached to deletes.

The second limit sits underneath every check in this chapter, because all of them treat the primary store as truth. The audit diffs the index against the chunk table, the query-time filter asks the document row which revision is current, and the reconciling writer trusts the chunks it is handed. Where the primary store is itself wrong — a chunker change altered boundaries, the chunk rows were rewritten under the new boundaries, and the document text was never reprocessed — a corrupted truth propagates cleanly into the index and every audit comes back clean. The counts agree, the revisions agree, and the content is wrong in both places. Only a comparison against the source document's text detects that, which is a different job with a different cost.

An update is two operations that every tool in the path presents as one. Upserting the chunks a document currently produces is the first. Removing the ids the previous revision produced and this one does not is the second, and nothing in the write path will ever remind you it is missing, because from the index's point of view every write it was asked to perform succeeded. Fourteen chunks became nine, five ids kept their old text, and every layer of the stack reported that the document had been updated.

The cost of doing this correctly is the subject of the next chapter. Reconciliation converts a silent orphan problem into a steady stream of real deletes: a corpus with 4% monthly edit churn now issues deletes for every shrinking revision, on top of whatever genuine document removals you already had. Those deletes do not free memory when they commit, and in a graph index they do not stop being traversed either, so the fix in this chapter feeds directly into a cost that most teams do not measure until their p99 latency has already moved.