A second ingestion path without the prefix never moves the average

Every chapter so far has assumed one corpus built by one pipeline, wrong or right in one uniform way. Corpora do not stay that way. A company that ships a retrieval system ships a second source into it within the year — a support-ticket connector, a wiki export, a backfill of archived PDFs — and that source is written by whoever owns the system it comes from, against the vector store's public interface, on their own schedule. The document prefix, the short literal string an asymmetric embedding model needs on the front of a passage so it is encoded in the passage role rather than the question role, is a rule that lives in the search team's code. The write happens somewhere else.

This chapter is about what that arrangement does to your ability to notice. The defect it produces is identical in kind to the ones in earlier chapters — raw text reaching the encoder where prefixed text should have — and completely different in how it presents, because it is partial. The arithmetic of a weighted average is what makes it invisible, and that arithmetic is worth doing before any of the guards make sense.

The arithmetic that guarantees a flat line#

There are two versions of this and both end in the same place. The first is that the labelled query set predates the connector. A set of 480 queries written against product documentation has relevance judgements naming chunks from the documentation corpus, and not one of them has a gold chunk that lives in a support ticket. A metric computed over them is measuring a part of the index the connector never touched, so it is not that the dashboard was insensitive — it was measuring something else entirely and reporting the number with the same confidence it always had.

The second version is the one that survives after somebody notices the first and adds queries for the new source. Suppose the set grows to 500 with 20 new queries covering support tickets, and suppose retrieval on those 20 collapses from the 0.79 the rest of the corpus enjoys down to 0.30.

The contribution column is the arithmetic, and it is worth reproducing rather than reading. A mean over 500 queries is the sum of each query's score divided by 500, which can be regrouped as each group's score multiplied by that group's share of the queries. The documentation group is 480 of 500, which is 0.96 of the set, and 0.96 times 0.79 gives 0.758. The support group is 20 of 500, which is four hundredths of the set, and four hundredths of 0.30 gives 0.012. Those two numbers add to 0.770, which is the reported average exactly, with nothing left over. That closure is what licenses the conclusion: there is no third term hiding in the metric, so the entire effect of the new source on the headline is the 0.012 line, and the headline moves from 0.790 to 0.770.

Two points. Whether two points is visible depends on what the metric does when nothing is wrong, and that has an arithmetic of its own. A recall figure over a labelled set is a proportion estimated from a sample of queries, so its standard error is the square root of the recall times one minus the recall, all divided by the number of queries. Near 0.78 on 500 queries that comes to about 0.019, and a conventional 95% interval is roughly two standard errors either side, so the interval is about ±0.037. A two-point drop sits comfortably inside the band where week-to-week resampling noise already lives.

The dashboard is not broken and the alert threshold is not badly chosen. The sensitivity of a weighted mean to any one group is exactly that group's weight, which is a property of the arithmetic and not of the system being measured. At 4% of the index — 168,000 chunks out of 4.2 million — a total failure on the affected documents produces a movement you could not distinguish from a quiet week, and no choice of threshold on that number can change it.

Every other signal in the system is similarly clean, and for the reason chapter 1 gave: each of them is a check on a quantity that does not depend on the prefix. Ingestion succeeded. The vectors have the right dimensionality and unit norm. Query latency is unchanged at 41 ms. The affected chunks are in the index, with correct text in their payload, retrievable by ID. Every one of those observations is accurate, and every one of them would read identically on a correctly built row and a raw one, because the only thing the prefix changes is where the vector sits. The only observable that would have shown anything is recall restricted to documents from that connector, and nothing in the system computed it.

From the outside, the failure reads as a coverage gap. Users say search does not know about tickets; the natural interpretation is that ingestion is behind, or that a filter is excluding them, or that the connector's backfill has not finished. Each of those has an obvious check that comes back green, which pushes the investigation towards "the model is not good at support tickets, they're messy and conversational" — the same conclusion that a formatting defect always suggests, arriving now with the extra plausibility of being about a genuinely different kind of text.

It is worth being precise about why a second ingestion path exists at all, because the guards below are aimed at that shape rather than at carelessness. Connectors get written by whoever owns the source system, on their own schedule, against the vector store's public interface. That interface accepts a vector and a payload and has no opinion about how the vector was produced, which is correct for a storage layer and is exactly the gap. The formatting contract lives in the search team's code, the write happens in the connector team's code, and there is no artefact in between that carries the rule. Nothing about that arrangement is unusual, and it will recur every time a new source is added.

Three guards, an hour in total#

The first is a choke point that makes the role impossible to omit. Every embedding call in the organisation goes through one function whose signature has no default.

from typing import Literal

Role = Literal["query", "document"]
PREFIXES: dict[Role, str] = {"query": "query: ", "document": "passage: "}

def embed(texts: list[str], *, role: Role) -> list[list[float]]:
    prefix = PREFIXES[role]                       # KeyError on anything else
    payloads = [prefix + t for t in texts]
    for p in payloads[:1]:
        if not p.startswith(prefix):              # cheap, catches upstream mangling
            raise ValueError(f"role={role} not applied: {p[:40]!r}")
    return _encode(payloads, normalize=True)

Keyword-only, required, and typed. A caller that forgets the role gets a TypeError at the call site rather than a silently unprefixed vector, which converts the entire class of defect from a quality regression into a build failure. The mechanism is worth naming: the defect exists because the role is a piece of information that can be absent without anything noticing, and a required argument is precisely a piece of information that cannot be absent without the program failing to run. This also removes the shared-helper default that produced the reversed case, because there is no default to be wrong about. A connector team would hit the type error on their first local run.

The second guard checks what is already in storage, on a schedule. It is the identity probe from chapter 2 applied to a random sample: pull rows, fetch their source text, re-embed with the document role, and compare against the stored vector.

def audit_stored_vectors(n: int = 200, floor: float = 0.999) -> list[str]:
    bad = []
    for row in index.sample(n, stratify_by=["source", "ingested_week"]):
        expected = embed([source_store.text(row.id)], role="document")[0]
        sim = cosine(expected, row.vector)
        if sim < floor:
            bad.append(f"{row.id} source={row.source} cosine={sim:.4f}")
    return bad

# 2026-04-22 audit:
#   chunk_9f30ab source=zendesk cosine=0.9612
#   chunk_1c77de source=zendesk cosine=0.9588
#   chunk_44b0e2 source=zendesk cosine=0.9634
#   ... 8 of 200 rows below floor, all source=zendesk

This is the guard that inverts the arithmetic from the start of the chapter, and the inversion is worth deriving because it is the least intuitive step here. An average over 200 sampled queries tells you nothing about a 4% defect, for the reason already established: the affected share multiplies into the result and 4% of anything is small. A per-row check over 200 sampled rows finds it almost certainly, and the reason is that it asks a different question of each draw. The probability that one uniformly sampled row is unaffected is 0.96, since 4% of rows are bad. Draws are independent, so the probability that all 200 are unaffected is 0.96 multiplied by itself 200 times, which is about 0.0003 — three chances in ten thousand of seeing nothing.

The sampler is strong exactly where the aggregate is weak, and the reason is that it never averages anything. Each row either matches its expected encoding or it does not, and one mismatch is a finding rather than a contribution to a mean. Averaging dilutes a signal in proportion to the share of the data it occupies; a search for any single instance grows more certain the more draws you take. Stratifying by source and ingestion week is what keeps that property useful as the corpus grows, since it guarantees a new source is covered on its first day rather than in proportion to its share of a growing index.

The third guard makes the metric capable of moving. Store the ingestion source in the payload, and have the evaluation harness report recall broken out by it rather than as a single number.

recall@10 by source, 2026-04-22
  docs           480 queries   0.79   (prev 0.79)
  runbooks        90 queries   0.81   (prev 0.80)
  zendesk         20 queries   0.30   (prev  n/a)   <- alert: new source below 0.60

Alerting per source rather than on the total is the whole change, and the operational half of it is a rule that a new ingestion path does not go live until it brings twenty labelled queries with it. That is an hour of work for whoever builds the connector and it is the only thing that makes their source visible to the evaluation at all.

The three are complementary rather than redundant, and the gaps line up against each other deliberately. The first prevents, and does nothing for the two million vectors written before it existed. The second detects what was already written, and compares against a source text that may itself have been rewritten since ingestion, which produces a false mismatch when it has. The third measures the thing users actually experience, and is blind to a source nobody wrote queries for. Each one's blind spot is inside another one's coverage, which is the only sense in which three guards are better than the best of them.

What the guards still let through#

Running the repair against a live index puts the corpus in a mixed state for six minutes, which for a repair of this size is usually acceptable — and the reason it is acceptable is that the affected rows are already returning bad results, so the transitional state is strictly better than the status quo rather than a regression from a working system. That reasoning does not hold for the reverse case, where a healthy index is being migrated to a new model, and the blue-green cutover from chapter 2 remains the right shape there. The same size of mixed window is fine in one direction and not in the other, which is a reminder that "how long is the migration" is the wrong question and "what is the state we are moving away from" is the right one.

The format_version filter is what makes the loop resumable. If it dies at 90,000 rows, the same command picks up where it stopped, because the rows already fixed no longer match the filter. Writing the version bump in the same upsert as the vector is what guarantees that property; bumping it in a separate pass leaves a window where a crash produces rows that are fixed but unmarked, and those get processed twice, which is harmless here and would not be for a job with side effects.

None of this makes formatting defects impossible. A team that changes model families and updates PREFIXES to the new strings without reindexing passes every one of these checks on the query side while the stored vectors carry the old family's roles. The choke point is satisfied, because a role was supplied. The per-source breakout will move, eventually, on whichever sources have labelled queries. Only the audit catches it directly, and only because it re-embeds rather than inspecting — which is a reminder that the audit is the guard doing the most work per line of code.

The guards also need an owner, and the honest answer is that the search team owns all three even though two of them constrain other teams' code. A connector team has no reason to care about a prefix, will not be paged when recall on their source drops, and cannot reasonably be expected to hold a rule that lives in someone else's model card. Shipping the choke point as a small internal package with the role argument required, and refusing writes that do not come through it, is the arrangement that puts the rule where the knowledge is. The alternative — a paragraph in an onboarding document asking connector authors to remember a string — has the failure rate you would expect.

What all of this adds up to is that the two prefixes stop being something a person has to remember at each call site and become a property the system enforces. The averages will keep being flat, because that is what averages do to a defect confined to one part of a corpus, and no threshold on a corpus-wide number will ever be sensitive to four per cent of it. What changes is that a required argument makes the omission impossible to write, a per-row audit makes it near-certain to be caught if it was written before the argument existed, and a per-source breakout makes it visible to the people who would otherwise be told their locale is simply hard.

Picking that up on a real service is a short sequence: open the card for the model you are actually running and copy the two strings exactly, write them into the index manifest at build time, put every embedding call behind one function that requires the role, log the final string on both sides at a low sample rate, and record today's per-source recall so that the next change has something to be compared against.