The invalidation key is a fingerprint of the text you embedded

The change was eleven lines inside extract_text(): drop <nav>, <header>, <footer> and any element carrying role="navigation" before flattening the remaining tree to plain text. It was a good change. The marketing site's chrome contributed roughly 60 tokens to every page — "Home Products Pricing Docs Blog Contact Log in Get a demo" — and those 60 tokens were sitting inside a mean 400-token chunk, taking up 15% of every vector the retriever ever compared a query against. The pull request was approved in twenty minutes and deployed on a Tuesday afternoon.

The nightly incremental job ran at 02:00 and produced this:

[02:00:04] incremental-embed start  watermark=2026-02-17T02:00:00Z
[02:00:09] candidate documents (updated_at > watermark): 0
[02:00:09] chunks staged: 0  chunks embedded: 0  tokens: 0
[02:00:09] done in 5.2s  estimated cost $0.0000

Nobody had edited a document in the CMS, so no updated_at had moved, so the job found nothing to do and said so cheerfully. Four weeks later a support engineer noticed that queries containing the word "pricing" were returning an almost arbitrary slice of the corpus, because a large fraction of the index still encoded a navigation bar in which the word "Pricing" appeared verbatim. The extractor had been fixed for a month. The index had never heard about it.

The index behind that search box is rebuilt, not built once#

The support engineer's complaint was about a search feature, and that feature works by comparing meaning rather than matching words. Behind it sits an embedding: a fixed-length list of numbers — 1,536 of them for the model in this system — that an embedding model produces from a passage of text, arranged so that passages about similar things come out numerically close to one another. Lists of that kind are called vectors, and a vector index is the store that holds one per passage and answers the question "which of these several million vectors sit nearest this one". A user's query is run through the same model, the index returns the passages whose vectors are closest, and a page about what the product costs comes back for the question "how much do I pay for five seats" even though the two share no words at all. That same closeness is how the navigation bar did its damage, because a chrome block containing the word "Pricing" dragged every page carrying it toward every question about price.

The stored units are passages rather than whole documents because an embedding model takes a bounded amount of text and compresses all of it into a single vector, so a forty-page manual encoded as one vector describes the manual and locates nothing inside it. A chunker therefore cuts each document into pieces of a few hundred tokens — the sub-word fragments a provider counts and bills by, averaging roughly four characters of English apiece — and each chunk is embedded and stored on its own. Getting a document from the system it lives in to a vector in the index takes a fixed sequence of steps, and that sequence is worth naming as a whole because this topic is about what it costs: fetch the record, extract text from the HTML or the PDF, normalise the result, chunk it, send the chunks to the provider, write the returned vectors into the index. That is the embedding pipeline, and you meet it as the batch job behind any feature that searches a company's own documents, whether that is site search, a support assistant, or retrieval that feeds a model's context.

The failure at the top of this chapter comes from treating the pipeline as preprocessing, something run once when the corpus is loaded, after which the index simply exists. Every stage of it is code that changes, sitting between source systems that change, calling a model that will eventually be withdrawn, so a corpus is embedded many times over the life of a system rather than once. Each of those passes carries a bill measured in tokens sent to the provider, a schedule set by the rate at which the provider will accept them, and a standing cost, since every vector occupies memory for as long as it remains searchable. Embedding pipeline economics is the practice of knowing which changes force which of those passes, what each will cost before it runs, and how to run one so that it survives being interrupted. Its cheapest and most common form is incremental re-embedding: recomputing vectors only for the chunks whose text has actually changed and leaving the rest of the index untouched. That is what the 02:00 job was written to do, and doing it correctly depends entirely on how the job decides what changed.

The key you select on decides what you can notice#

An incremental re-embed is a cache invalidation problem wearing different clothes, and the whole behaviour of a cache is determined by the key. Selecting on updated_at encodes a specific claim: that the only thing which can change the text sent to the embedding model is an edit to the source record. That claim is false in every pipeline that does any processing between the source and the model, which is every pipeline. The text sent to the model is a function of the source record and the extractor and the normaliser and the chunker and whatever prefix or instruction the model requires. A change to any of those five inputs changes the output, and only the first of them has a timestamp attached.

The fix is to invalidate on a fingerprint of the exact bytes handed to the provider, combined with an identifier for the configuration that produced them. Two pieces of state make that work. The first is a pipeline id, a short hash over the canonical serialisation of every version and parameter that can influence the text or the vector. The second is a chunk fingerprint, a hash over the chunk's final text together with that pipeline id.

import hashlib
import json

PIPELINE = {
    "extractor": "html2text@2026.02.1+strip_chrome",
    "normaliser": "nfkc_collapse_ws@3",
    "chunker": {"name": "recursive", "size": 512, "overlap": 0.10, "unit": "tokens"},
    "model": {
        "id": "embed-4-large",
        "snapshot": "2026-01-22",
        "dim": 1536,
        "input_prefix": "passage: ",
    },
}


def pipeline_id(cfg: dict) -> str:
    canonical = json.dumps(cfg, sort_keys=True, separators=(",", ":"))
    return hashlib.blake2b(canonical.encode("utf-8"), digest_size=8).hexdigest()


def chunk_fingerprint(chunk_text: str, cfg_id: str) -> str:
    h = hashlib.blake2b(digest_size=16)
    h.update(cfg_id.encode("ascii"))
    h.update(b"\x00")
    h.update(chunk_text.encode("utf-8"))
    return h.hexdigest()

Two details in that dictionary are load-bearing. snapshot is pinned rather than left to a floating alias, because providers do rotate the model behind an unversioned name, and a vector produced by the January weights and a vector produced by the April weights are not comparable even when the string embed-4-large is identical in both cases. input_prefix is in there because asymmetric models require documents and queries to be embedded with different instructions, and the day someone corrects passage: to search_document: is a day the entire index becomes stale without a single source record changing. Anything that alters the bytes crossing the network belongs in the pipeline id. Anything that does not — the number of worker threads, the retry policy, the log level — must stay out, or you will invalidate 8 million vectors because someone raised a timeout.

Deciding which side of that line a dependency falls on is the part that needs care rather than a rule. The extractor's version string covers the extractor, but the extractor imports an HTML parser, a Unicode normaliser and a tokeniser, and a patch release of any of them can change output — a lxml bump that starts preserving whitespace inside <pre>, a normalisation library whose handling of zero-width joiners changes, a tokeniser update that splits emoji differently. The pragmatic approach is to include the resolved versions of the handful of libraries that touch text, taken from the lockfile at build time, and to accept that this over-invalidates occasionally. Over-invalidating costs a re-embedding run you did not strictly need; under-invalidating costs a month of quietly wrong retrieval, and the two are not symmetric.

With those two values stored beside each vector, the nightly job stops asking a question about time and starts asking a question about content:

-- what ran, and found nothing
SELECT id FROM documents WHERE updated_at > :last_run;

-- what should run
SELECT c.chunk_id, c.text
FROM staged_chunks c
LEFT JOIN vectors v USING (chunk_id)
WHERE v.chunk_id IS NULL
   OR v.fingerprint <> c.fingerprint;

Run against the corpus on the Wednesday morning, the second query would have returned 1.14 million chunks out of 1.2 million — every chunk from every page that had chrome — and the job would have re-embedded them for about $9 at $0.02 per million tokens. The cost of noticing was never the problem. The problem was that nothing asked.

Fingerprinting everything is expensive; gate it in two tiers#

The obvious objection to content-addressed invalidation is that computing the fingerprint requires producing the text, and producing the text requires fetching, parsing and chunking the source. On a 1.2 million document corpus with a mean extraction time of 12 ms, a full sweep is four CPU-hours; on the 8 million document corpus this topic returns to repeatedly, it is 26 CPU-hours per nightly run, spent almost entirely to discover that nothing changed. That is real money and a real scheduling problem, and it is why teams reach for the timestamp in the first place.

The way out is to keep the timestamp as a cheap upstream gate rather than as the authority. A source content hash — or the CMS's own updated_at, or an ETag, or a change-data-capture stream, meaning a feed of row-level changes emitted by the source database as they commit — decides whether the expensive extraction runs at all. The chunk fingerprint decides whether the expensive API call runs. When the pipeline id changes, the upstream gate is bypassed for everything, because a pipeline change is by definition a change that source metadata cannot see.

The second gate earns its place even on a full sweep. When the extractor changed, every document had to be re-parsed, but only the pages that actually carried chrome produced a different fingerprint; the 60,000 API-generated documentation pages rendered without a navigation wrapper hashed identically and cost nothing. A pipeline change forces a full pass over the corpus, and the fingerprint decides how much of that pass reaches the provider. On the chrome-stripping deploy that was 95%, which is close to a full reindex. On a normaliser change that only affects documents containing non-breaking spaces, it might be 3%.

Different keys catch different classes of change, and it is worth being explicit about which:

Adopting the last row on an index that already exists has one awkward step, which is that you do not know what text produced the vectors already in the store. Nothing recorded it. The honest options are to re-embed the whole corpus once and record fingerprints as you go, which is expensive and gives a known-good baseline, or to run the pipeline over every document, record the fingerprint of the text it produces now, and write it against the existing vectors as an assumption. The second costs only CPU and is usually the right call, provided everyone understands what it asserts: that today's pipeline would have produced today's vectors. Any drift that happened before the fingerprints were written stays invisible, so the assumption is worth pairing with a spot check — embed a random 2,000 chunks fresh, compare them against the stored vectors by cosine similarity — the angle-based closeness score, running from 1 for identical direction down through 0 for unrelated, that the index itself ranks by — and see how many come back below 0.999. On a healthy index that number is zero; on the corpus in this chapter it would have been 95%.

"Partially" in the third row is the interesting cell. A hash of the whole extracted document notices that the document's text changed, which is enough to decide whether to re-embed that document, but it cannot tell you which of its chunks changed, so it re-embeds all of them. On a 40-chunk document where one sentence was edited, that is 40 API calls where 2 were needed. Fingerprinting at chunk granularity is what turns a document edit into a proportionate amount of work, and on a corpus where edits are small and documents are long, the difference is a factor of ten in spend.

What a fingerprint does not buy you#

It does not tell you what changed or why, only that something did. A pipeline id that flips from a3f1c2 to 9d40be invalidates everything downstream of it with no indication of which field moved, so the id belongs in a table beside the serialised config it was computed from, and every deploy that changes the config should write a row. Without that, the six-month-old question "why did we re-embed the whole corpus on 3 March" has no answer.

It does not bound the blast radius, either. Fingerprinting is a mechanism for detecting staleness, and detection is orthogonal to scope. Some pipeline changes touch a handful of documents and some touch all of them, and the fingerprint reports both the same way — as a set of chunk ids that need work. The next chapter is about the case where that set is the entire corpus by construction, and about why "we'll just re-embed the affected subset" is a sentence that quietly stops meaning anything.

It also does not detect the kind of staleness where the text and the configuration are both unchanged and the index is still wrong for the traffic — vocabulary drifting away from what the model was trained on, or a corpus whose subject matter has moved. Index maintenance covered that; a content hash is blind to it by design, because nothing in the bytes has moved.

The most tempting misuse is to fingerprint the source and call it done, because it is cheap, it is one line, and it catches the failure everybody imagines: someone edits a document. The failures that actually leave a corpus stale for a month come from the other direction, where the documents sit perfectly still and the code around them moves. A deploy that changes a parser is a change to every vector in the index, and the only place that fact is visible is in the bytes you handed to the model.

The mechanism is now in place to answer the question "which chunks need re-embedding?" for any change at all. What it does not answer is the question a finance director asks, which is how much of the corpus a given change will touch before you run it — and for one very common class of change, that answer is all of it.