Deterministic chunk ids turn a resume into a set difference

The pod was OOM-killed at 09:14 — terminated by the kernel for asking for more memory than its limit allowed — nine hours into a twelve-hour run. There was no checkpoint file, because the job had been written to stream: read documents from a cursor, chunk, pack, embed, upsert, move on. The vectors it had written were in the store and serving. The on-call engineer's question to the channel was straightforward — can we pick up where it left off, or does this start again from zero — and the answer had been fixed months earlier, in the four lines that decided what a chunk's identifier is.

The job assigned identifiers with uuid4(). Nothing in the store could be asked which chunks were already done, because the ids in the store bore no relationship to any id the next run would generate. Restarting was the only option, and restarting meant re-embedding the 4.5 million chunks already present, generating 4.5 million new vectors with new random ids alongside the old ones, and finishing some time on Sunday.

The only question the store needs to answer#

A resume is possible exactly when you can compute the set of work items that remain, which requires a common vocabulary between the plan and the store. The plan knows which chunks it intends to write. The store knows which chunks it holds. If those two are expressed in the same identifiers, the remaining work is planned − present, and computing it is a lookup. If they are not, the store's contents are opaque, and the only safe assumption is that nothing is there.

A deterministic identifier gives that shared vocabulary. It is derived from the document identity, the chunk's position within the document, and the configuration that produced the chunk boundaries — the text-producing half of the pipeline identity that chapter 2 separated out.

import hashlib
import uuid

def chunk_id(doc_id: str, ordinal: int, text_config_id: str) -> uuid.UUID:
    h = hashlib.blake2b(digest_size=16)
    h.update(doc_id.encode("utf-8"))
    h.update(b"\x1f")
    h.update(str(ordinal).encode("ascii"))
    h.update(b"\x1f")
    h.update(text_config_id.encode("ascii"))
    return uuid.UUID(bytes=h.digest(), version=4)

The separator byte between fields is not decoration. Concatenating doc_id and ordinal directly makes ("page-1", 12) and ("page-11", 2) produce identical input, and on a corpus with numeric document ids that collision is not hypothetical. Wrapping the digest as a UUID is a concession to vector stores that accept only UUIDs or unsigned 64-bit integers as point ids; the 128 bits survive intact.

Digest width is worth one line of arithmetic rather than a shrug. At 400 million chunks, a 128-bit identifier has a collision probability of about 2.4 × 10⁻²², which is not a thing that happens. A 64-bit identifier at the same scale has a collision probability of roughly 0.4%, or one chance in 230 that two distinct chunks silently occupy the same slot and one of them is invisible to retrieval forever. Truncating a hash to fit an integer column is the kind of decision that looks free and has a birthday bound attached — the arithmetic by which collisions become likely once the number of items approaches the square root of the id space rather than the space itself.

The third row is the one that looks most appealing and causes the most damage. Content-addressed ids give perfect deduplication, which sounds like a feature until a legal disclaimer appearing verbatim in 200,000 contracts becomes a single point in the index, attributable to one arbitrary document, and every query that matches it returns that one document. Deduplicating identical text is a genuine optimisation — it can remove 5 to 15% of a corpus's tokens on document sets with heavy boilerplate — but it belongs in a separate mapping from text hash to the list of documents that contain it, not in the identity of the point.

The fourth row fails in the opposite direction: the id is stable across a re-chunk, so chunk 3 of document X after the change overwrites chunk 3 of document X from before, even though the two carry different text and there are now nine chunks where there were four. The leftover chunks 5 through 9 from the old configuration have no writer and are never cleaned up. Including the text config in the derivation makes a re-chunk produce an entirely new id space, which is the correct behaviour for a change that chapter 2 established has no affected subset.

The resume, as an anti-join#

With deterministic ids, the recovery at 09:14 is a query. The planner regenerates the full list of intended chunk ids — which is cheap, because it needs document ids and lengths rather than document text — and asks the store which of them are present with a current fingerprint. The query is an anti-join, meaning it returns the rows of the plan that have no matching row on the other side, which is exactly the definition of remaining work.

-- staged_chunks holds (chunk_id, fingerprint) for the run's full plan
SELECT s.chunk_id
FROM staged_chunks s
LEFT JOIN vectors v
       ON v.chunk_id = s.chunk_id
      AND v.fingerprint = s.fingerprint
WHERE v.chunk_id IS NULL;

Where the vector store is not the sort of thing you can join against, the same anti-join runs client-side in batches, and the cost is small enough to be uninteresting: 6 million ids at 1,000 per existence check is 6,000 round trips, and at 30 ms each that is three minutes. A full scroll — a paged sweep that pulls every id the store holds, a page at a time — is often faster still, on the order of 50,000 ids per second, so two minutes for 6 million. Either way the resume decision costs single-digit minutes against a nine-hour alternative.

Regenerating the plan is cheap because it does not need document text. Chunk ordinals come from a token count, and a token count per document is something the pipeline already stored the first time it ran, so staged_chunks can be rebuilt from a table of document ids, token counts and the text config id in a few minutes of SQL. Where that table does not exist, the resume degrades to re-reading and re-chunking the corpus — CPU rather than tokens, which on 8 million documents at 12 ms each is a few hours on a modest worker pool and still far cheaper than re-embedding. Keeping the token count alongside the document is the small piece of bookkeeping that keeps the expensive path from being necessary.

A watermark still has a role in this design, as an optimisation rather than as the authority. The anti-join over the whole corpus is correct and takes minutes; the anti-join over documents whose source hash changed since the last successful run is equally correct in the common case and takes seconds. Running the cheap version nightly and the full version weekly gives both properties, and the weekly run is what catches anything the cheap version's assumptions missed.

That alternative is what makes the property worth designing for. Restarting the job means re-sending the 4.5 million chunks that were already embedded, which is 1.8 billion tokens, $180 at $0.10 per million — a number small enough that nobody would notice it on an invoice. The expensive resource is quota-time. Those 1.8 billion tokens occupy nine hours of a shared 5 million tokens-per-minute allocation, during which the production ingest path is competing for the same limiter and new documents are not reaching the index. The cost of a restart is measured in hours of throughput that cannot be bought back, plus the fact that the twelve-hour window everybody agreed to has become twenty-one hours and now spans a Monday morning.

Duplicates are the failure that does not announce itself#

The second consequence of random ids is quieter than the lost time. Re-running the job writes a second vector for every chunk it already wrote, because there is no key on which an upsert — a write that inserts a point if the id is new and replaces the point already sitting there if it is not — can match. The store's point count goes from 4.5 million to 10.5 million on a job that was supposed to produce 6 million, and nothing errors.

What that costs, using the per-chunk figures from chapter 4, is 4.5 million surplus points at 1,872 memory-resident bytes across three replicas, which is 25 GB of RAM and about $101 a month, continuing until somebody works out where the extra points came from. The retrieval cost arrives sooner. A query whose top ten would have contained six distinct documents now returns three documents twice each, the reranker — the second-stage model that rescores retrieved candidates before they are shown — sees redundant context, and recall@10 measured against a fixed judgement set, a list of queries with the documents a human has marked relevant to each, drops by several points for reasons that look like a model problem.

Deterministic ids make the write idempotent, so a re-run of any subset is harmless by construction: the same input produces the same id, the upsert replaces the point in place, and running the whole job twice produces exactly the state that running it once produces. This is the property that makes at-least-once delivery acceptable — delivery under which an item may be processed more than once and is never processed zero times — and at-least-once is the only delivery guarantee available across a network boundary that includes a client timeout. It pairs with request-level idempotency keys where the provider supports them, which is what keeps a resubmitted batch after a client-side timeout from being billed twice — the few percent that chapter 3 budgeted as a partial-failure allowance.

Where determinism stops helping#

It does not tell you whether a vector is correct, only that a point with that id and fingerprint exists. A run that was writing garbage — a bug in the extractor, a truncated text field — writes it under perfectly stable ids, and the anti-join will happily declare that work done. Resumability and correctness are separate properties, and the second one needs the fingerprint to actually be computed from the text that was sent rather than from the text that was intended.

It also does not handle removal. Documents deleted from the source leave points whose ids no longer appear in any plan, and an anti-join only ever finds work to add. Reconciling in the other direction — finding points the plan no longer contains — is the sweep that index maintenance covers, and it has to run separately, on its own schedule, against its own definition of authoritative.

And a deterministic id leaks its inputs into an interface. Once external systems store chunk ids, changing the derivation — adding a field, switching hash functions, correcting the separator — invalidates every reference, so the derivation deserves a version prefix from the first commit and the same treatment as a public API afterwards.

A backfill that is packed to a token budget, throttled to a share of quota, and resumable from an anti-join is a well-behaved job for any single change to a corpus. Applying the same machinery to a change of embedding model turns out to be a different exercise, because for the first time the old vectors and the new ones cannot be compared, mixed, or served from the same place, and the cost stops being a function of how many tokens you send.