Deterministic chunk keys turn a redelivery into an overwrite

A document does not go into a vector index whole. It is split into chunks, passages of a few hundred tokens each, and every chunk becomes one row in the index: an identifier, a vector of numbers standing for the passage's meaning, and some metadata. Writing a document therefore means writing a set of rows, and something has to decide what identifier each of those rows carries. That choice looks like a formality, and it settles what happens the second time the same document arrives.

Documents do arrive twice. A connector that commits its cursor after the upsert, which is the arrangement chapter 1 argued for on the grounds that the alternative loses documents permanently, guarantees that a crash replays whatever it was working on. A broker with at-least-once delivery guarantees the same thing from the other direction: it re-offers any message that was not acknowledged in time. And so does a poll window deliberately overlapped by 30 seconds to avoid skipping documents whose timestamps tie on a one-second boundary. Redelivery is therefore designed into such a pipeline at three separate layers, and none of the three is a bug to be removed. The only question worth asking is what the index looks like after a document has been delivered for the second time.

Where a row's identity comes from#

The failure has one cause: each attempt minted new row identities. The vector client's upsert is genuinely an upsert — it replaces the row at a given ID, creating it if no row is there — but an ID drawn from uuid4() never collides with the previous attempt's ID, so upsert degenerates into insert. The same thing happens with str(uuid.uuid5(NAMESPACE, str(time.time()))), with autoincrement primary keys, and with any scheme where the identity of a row is produced by the act of writing it rather than derived from what is being written.

That last sentence is the general rule, and it is worth walking rather than asserting. An upsert is defined relative to an address: given an address, write this content there, replacing whatever was there before. The behaviour a reader wants — the second write cancelling the first — requires two attempts to compute the same address. Whether they do is decided entirely by the function that produces the address. uuid4() reads from a random source and is specified to produce a fresh value every call, so two attempts cannot agree. An autoincrement column derives its value from the database's counter, which advances on every insert, so two attempts cannot agree there either. Hashing the current time gives a function whose input differs between attempts by construction. In each case the address is a function of when the write happened, and the two writes happened at different times, so the addresses differ and both rows survive.

This is where the belief that does the damage sits, and it is worth naming because it is genuinely reasonable. A team that has read about at-least-once delivery knows the defence is idempotent writes, looks at its own code, sees index.upsert(...), and concludes the defence is in place. The method is called upsert. The documentation says it overwrites. Both of those are true.

What makes the belief false is that the quantity being checked stays constant while the dangerous property varies. The call is index.upsert(...) in the broken version and index.upsert(...) in the fixed version — same method, same client, same log line, same metric. Everything a reviewer would look at is identical across the two designs. The property that actually differs is one level below: whether the id field handed to that call is a function of the document or a function of the moment. An overwrite is a property of an address, and asking whether the code calls upsert cannot distinguish an address that repeats from one that does not. The check that does distinguish them is to compute the key for the same chunk twice, in two processes, and compare the two strings.

A key derived from what is being written#

The repair is to make the identity a pure function of the document and the position of the chunk within it, so that any attempt to write chunk 3 of handbook/pto-policy targets the same row as every previous attempt:

import hashlib

def chunk_id(source: str, doc_id: str, chunk_index: int) -> str:
    # Stable across processes, deploys, retries and machines.
    raw = f"{source}\x1f{doc_id}\x1f{chunk_index}".encode("utf-8")
    return hashlib.sha256(raw).hexdigest()[:32]

def write_document(doc, chunks):
    rows = [
        {
            "id": chunk_id(doc.source, doc.id, i),
            "values": embed(text),
            "metadata": {
                "source": doc.source,
                "doc_id": doc.id,
                "chunk_index": i,
                "source_version": doc.etag,
            },
        }
        for i, text in enumerate(chunks)
    ]
    index.upsert(rows)

Every input to chunk_id is a property of the document, and none of them is a property of the attempt. That is the entire content of the fix: there is nothing in that function a second process running an hour later on a different machine could disagree about, provided it is handed the same document.

The hash is there for hygiene rather than for uniqueness: it gives a fixed-length key that is safe in a URL, a Pinecone ID and a Postgres primary key at once, and it stops the raw document path from leaking into logs and IDs. A plain f"{source}:{doc_id}:{i}" is equally correct where the character set and length allow it, and is easier to debug because you can read a row's provenance off its key. The \x1f unit separator matters more than it looks: concatenating source + doc_id with a delimiter that can appear inside either field lets ("wiki", "a:b") and ("wiki:a", "b") collide, and a collision here silently overwrites one document's chunk with another's.

That collision deserves to be separated from everything else in this chapter, because it points the other way. Every other failure here leaves rows in the index that should not be there — extra copies, stale revisions, orphans. A delimiter collision destroys a row that should be there, by writing one document's content over another document's address. Accumulation is recoverable by deleting; a destroyed row is recoverable only by re-ingesting the document that lost it, and nothing in the index records which document that was. Choosing a separator that cannot occur in a path is a one-character decision guarding the one failure in this chapter that loses data.

What the row count converges to#

Take a document that splits into nine chunks and is delivered three times: a first attempt that gets four chunks into the index before its embedding call fails, and two later attempts that run to completion. Chunks are numbered from zero in the code above and from one in ordinary speech, so the four chunks of that first partial write are chunks 0–3 by index and chunks 1 through 4 by count; they are the same four passages.

With deterministic keys the sequence resolves. The first attempt writes rows for chunks 0–3 and fails. The second attempt writes rows for chunks 0–8, where the first four are overwrites of rows that already hold exactly that content and exactly that vector, and the last five are inserts. The third attempt overwrites all nine. Total rows in the index for that document: nine, which is the number the document produces. The redundant work is real — the pipeline paid to embed 22 chunks to store nine, which at roughly 400 tokens per chunk and $0.02 per million input tokens is a rounding error per document and a visible line item across 4M of them — but the state it leaves behind is correct.

The count of 22 is worth checking from both ends, because it closes exactly and that tells you the history is fully accounted for. Counting by attempt, the three writes were of four rows, nine rows and nine rows, and those add to twenty-two. Counting instead by chunk, in the version without deterministic keys where nothing overwrites anything, the first four chunks end up stored three times each and the remaining five twice each, and those two products add to twenty-two as well. Two routes that share no step — one walking the timeline, one walking the document — land on the same number, which licenses a specific claim: every row in that index is attributable to exactly one attempt, and there is no fourth delivery or unexplained partial write hiding in the total. What it does not license is any claim about content. All twenty-two rows hold the same nine passages, which is precisely why the fix is a change of key rather than a change of what gets written.

The general statement behind the worked case is short, and it is the reason determinism is worth the trouble. Let n be the number of chunks a document produces and let there be any number of delivery attempts, complete or partial. With keys derived from the document, every write lands inside the fixed set of n addresses, so the number of rows is at most n, and after any one complete attempt it is exactly n — independent of how many attempts there were. With keys minted at write time, the number of rows is the sum of the chunks written across all attempts, which grows with every retry and has no upper bound at all. Same delivery history, same content, same upsert call; one row count is constant in the number of attempts and the other is linear in it.

The cost of the linear one is paid at query time, and it is paid in slots. A retrieval system fetches some fixed number of candidates — a top-N window — and hands them on. If one passage occupies k of those slots, then k − 1 of them are carrying text the reader has already seen, and the same number of distinct passages that would otherwise have made the window are pushed out of it. The window size never changes and nothing errors; the number of different things in it falls by exactly the number of redundant copies.

Choosing what counts as the same document#

The chunk index is trivial. Everything difficult about deterministic keys lives in the choice of what doc_id means, because that choice decides which real-world events count as "the same document" and which count as a new one.

The table is most useful read as the gaps between rows rather than as five verdicts. The path row and the opaque-ID row differ in one column only, the rename column, and that single column is what a folder reorganisation exercises and what no test suite reproduces. The opaque-ID row and the content-hash row differ in two columns, and both differences have the same cause: a hash is computed from the body, so it cannot be stable across an edit and cannot be computed at all for a document whose body you no longer have. The last column of the table is the same observation stated as a consequence.

The title row is the only one whose named failure is a collision rather than an accumulation, which puts it in a different category from the other four. Losing a row and gaining a stale one are not degrees of the same problem.

The composite row's entry in that column is a conditional, and the condition is doing all the work. "Nothing, when the source has a real ID" means that a composite key inherits the failure of whatever was substituted for a real ID — pair a connector name with a path and you have the first row back, with an extra prefix.

A source-assigned opaque ID — Confluence's pageId, Google Drive's fileId, a database primary key, S3's key plus a version-independent object identity — is the right answer whenever it exists, and it exists more often than teams check. Paths are what teams reach for because they are visible in the crawler, and the cost lands weeks later when somebody reorganises a folder and 2,300 documents re-ingest under new identities while their originals stay indexed under the old ones.

A content hash as a change detector#

The content-hash row is worth dwelling on because it is used correctly for something else. Hashing the document body gives you a change detector: store the hash alongside the record, and on the next sync compare the incoming hash against the stored one to decide whether the document needs re-parsing and re-embedding at all. LlamaIndex's ingestion pipeline builds exactly this, with a docstore holding doc_id -> hash and an upsert strategy that skips unchanged documents, and its "Ingestion Pipeline + Document Management" example is the shortest runnable demonstration of the pattern. Using the hash as the key rather than as the comparison value is the mistake; it makes revision 5 of a document a different entity from revision 4, so the index accumulates one copy per revision and never learns that the old ones are dead.

The distinction is the same one the whole chapter turns on. A key answers "which row is this", and the answer must not change when the content does, or the system loses track of what it already stored. A change detector answers "is this the same content", and its answer must change when the content does, or it never fires. Those are opposite requirements on the same hash, which is why one value cannot do both jobs, and why a design that uses it for both looks self-consistent right up to the first edit.

The saving from a hash comparison is substantial on a source that reports changes generously. Sources whose updated_at moves when a page is viewed, when a label is applied, or when a parent page is edited will hand a connector far more "changed" documents than have actually changed. On the 4M-page wiki, roughly 6% of pages are reported as modified daily, which is 240,000 documents, and a hash comparison typically finds that 40–60% of those have identical bodies. Skipping them turns about 96M tokens of embedding work per day into about 48M, and more usefully halves the load on the parser, which is where the wall-clock time actually goes.

That chain is worth following through, since every step is a multiplication and the last one is a midpoint rather than a bound. A fraction of the wiki reported as modified gives the daily document count. Dividing the daily token figure by that document count gives the tokens each reported-modified document costs to embed, and it works out at the same few hundred tokens that one chunk carries — which holds only if a typical wiki page is about one chunk long, and is the figure to re-derive on a corpus of longer pages before quoting either number. Halving is the middle of the band: at the pessimistic end of it, two-fifths of the documents are skipped rather than half, and at the optimistic end three-fifths are.

The residual is in the word "skipping", and it is the step the arithmetic hides. To compare a content hash you must have the content, which means the connector still fetches all 240,000 bodies before it can decline to embed any of them. The saving is on embedding and parsing, and there is none at all on the fetch. The metadata in the code above points at the way out: source_version holds the document's ETag, a value the source computes and returns in a listing, so a connector that can compare versions without downloading bodies skips the fetch too. Whether it can is a property of the source's API rather than of the pipeline, so it is worth checking before a resync is budgeted.

What the chunk index assumes about the splitter#

The chunk index carries a quieter assumption, which is that the splitter is deterministic. A fixed-size or sentence-boundary splitter run twice over identical bytes produces identical boundaries, so chunk 3 means the same span on both attempts and the overwrite is a true no-op. A splitter that consults a model — a semantic splitter that embeds candidate boundaries, or an LLM-based structural splitter with a non-zero temperature — can return eight chunks on one attempt and nine on the next for the same input, at which point a redelivery writes different text into the same keys and leaves a trailing orphan.

The orphan is a direct consequence of how the key is built. Addresses are generated by counting up from zero to one short of the chunk count, so a run producing fewer chunks generates a shorter list of addresses, and the rows at the addresses beyond the end of that list are never written to and never deleted. They keep whatever the longer run put there. The index then holds a document whose passages come from two different splits, and neither the row count nor any error rate reports it.

If the pipeline uses a non-deterministic splitter, the chunk boundaries have to be persisted alongside the document and reused on replay, or the whole document has to be rewritten under the generation-and-sweep discipline that chapter 3 describes. Pinning the splitter's version and its parameters in configuration, and treating a change to either as a corpus-wide event rather than a deploy, is the cheap half of that discipline.

Idempotency keys, POST and PUT#

The pattern here is the idempotency key, which payment APIs standardised long before anybody was embedding anything and long before vector databases existed. A client sends Idempotency-Key: 4f2c… with a charge request; the server records the key with the outcome; a retry carrying the same key returns the original outcome instead of charging the card twice. HTTP encoded the same distinction in 1997 in the difference between POST and PUT: POST /documents creates a new resource on every call, and PUT /documents/pto-policy addresses one resource whose state converges to whatever the last successful call said, however many times you send it.

Both of those are comparisons rather than descriptions of what a vector index does, and each breaks in a place worth knowing. A payment server stores the outcome against the key and replays it, so a retried charge does no work at all. A deterministic chunk key stores no outcome; the retry runs the parse, runs the embedding, and writes rows whose content happens to equal what was already there. What determinism buys is convergence, not suppression, and the twenty-two embeddings paid for above are exactly the difference between the two. A pipeline that wants suppression as well needs the change detector from the section above, which is a separate mechanism doing a separate job.

The PUT comparison breaks somewhere else. A PUT addresses one resource, and a document is not one resource here — it is n rows, so the write is n PUTs issued together. Nothing in that set addresses a row the current split does not produce, which is why a document that shrinks leaves rows behind. The analogy holds for as long as the number of chunks stays fixed and fails at exactly the moment it changes.

An ingestion pipeline is a PUT and it is very often written as a POST. The reason is that the language of the tooling encourages it — you have a list of chunks, you call add() or add_documents() or insert(), and the library generates IDs for you because it has no idea what your documents are called. The default is genuinely the right default for a library that cannot know your identity scheme, and it is the wrong default for every pipeline that runs more than once.

Deduplication after the fact#

The alternative that teams reach for when duplicates appear is deduplication after the fact: a nightly job that groups rows by content hash and deletes all but one, or a near-duplicate filter applied to retrieval results before they reach the reranker. Both work, in the sense that the symptom goes away. The nightly job has to scan the whole index, it cannot run inside the write path, and it leaves a window of up to 24 hours where retrieval quality is degraded. The query-time filter pays a similarity comparison on every result of every query in order to compensate for a mistake made once at write time, and it hides the duplication from your own metrics, so you stop finding out that the write path is broken. Neither addresses the case where the two copies are not identical, which is the subject of chapter 3.

That last clause about metrics is the one to weigh most heavily, and it is the same shape as the green dashboards of chapter 1. A query-time filter removes duplicates from what you look at while leaving them in what you store, so the quantity a team watches stops responding to the quantity that is actually moving. The write path can degrade from a few duplicated documents to most of them without a single graph changing.

Late writes, shrinking documents and deletions#

A deterministic key makes a repeated write harmless. It does nothing about a late write, because the row is addressed by identity rather than by version, so an old payload arriving after a new one overwrites the new one perfectly correctly according to the rules the system was given. It does nothing about a document that used to produce nine chunks and now produces five, which is the orphan problem topic 8.17 covered from the index-maintenance side, where stale rows left in an index keep being retrieved, and which chapter 3 revisits as an obligation on the writer to remove what its own shorter split no longer addresses. And it does nothing about deletion, since a deleted document produces no write at all.

Those three limits share a cause. The idempotency a deterministic key provides is a property of a write's address: it fixes which row a write lands on, and says nothing about whether that write should happen, in what order, or on behalf of rows nothing is writing to. Ordering, versioning and absence are three separate mechanisms, and each needs something the key does not carry.

What it does buy, beyond correctness under retry, is the answer to the objection raised at the end of chapter 1 — that committing the cursor late is the worse design because it reprocesses documents and creates duplicates. With deterministic keys, the reprocessing produces no duplicates at all; it produces repeated writes of identical rows, which cost compute and change nothing. Even without them, duplicates remain a countable, addressable state:

SELECT doc_id, chunk_index, count(*) AS copies
FROM chunks
GROUP BY doc_id, chunk_index
HAVING count(*) > 1
ORDER BY copies DESC
LIMIT 50;

That query runs in seconds, tells you the exact blast radius, and names the documents to rewrite. There is no equivalent query for the gap that the early commit leaves, because the rows that would answer it were never written and nothing recorded that they should have been. The two failure modes differ in whether the evidence exists, and that difference is what settles the argument.

Every mechanism in this chapter is one decision applied consistently: the address a row lives at is computed from the document, never from the attempt. That single property is what turns a redelivery into an overwrite, holds the row count at the chunk count however many times a message is replayed, and lets the query above return an empty result rather than a work list.

Ordering is the part this leaves open. A pipeline that retries, backfills and replays will deliver the same document more than once, and the copies will not always carry the same revision.