A half-normalised index ranks by magnitude, not relevance

A production corpus is rarely written by one piece of code. There is usually a bulk job that loaded the initial documents and a worker that has been upserting new ones ever since — inserting each document, or overwriting it in place when the index already holds that identifier — and the two were written at different times by different people. Nothing in a vector database's configuration notices when those two paths disagree about whether to normalise before writing, because the database accepts any array of the right width.

That disagreement is harmless under some metrics and catastrophic under others, and which one you get depends on a setting made elsewhere. Under dot product it produces a failure with an unusually clean shape, which matters because of how the report arrives: a recall figure that dropped after a migration with nothing else touched has several candidate causes, and this one is the cheapest of them to rule in or out.

Two ingest paths, one collection#

Take a corpus of fifty-two thousand documents that arrived through two different pieces of code. A bulk loader written for a migration, whose author read the previous system's setup carefully enough to notice that it normalised before adding vectors to an IndexFlatIP — FAISS's exhaustive inner-product index — so the loader called model.encode(batch, normalize_embeddings=True). And an incremental upsert worker that was not written for the occasion; it had been running for months, feeding new documents into the old index through a helper that normalised in a separate step further down the call stack — a step that did not survive the port, because the port only replaced the part of the function that talked to FAISS.

Under dot product, a document's score is its norm multiplied by the cosine of its angle to the query, which is the geometric form of the dot product from the first chapter with the query's own norm set to 1. Work one pair of candidates through it. An unnormalised document with norm 10.7 and a mediocre cosine of 0.42 scores 4.49, since 10.7 multiplied by 0.42 is a little under four and a half. A perfectly normalised document with an excellent cosine of 0.83 scores 0.83, because multiplying by a norm of 1 leaves the cosine where it was. The unnormalised one wins by a factor of five while being roughly half as relevant.

The important part of that comparison is not the size of the gap but its robustness. Cosine is bounded above by 1, so the very best a normalised document can score is 1. The unnormalised population starts at a norm of 6.2, so the very worst any of them can score against a query they are even loosely aimed at is well above that ceiling. The two populations therefore do not overlap in score at all: every unnormalised document beats every normalised one for every query, regardless of what any of them say. That is why the effect is uniform across the corpus rather than concentrated in hard queries, and it is the fact the diagnostics below are all measuring.

The aggregate recall figure hides that structure, and splitting it by ingest path exposes it immediately. Queries whose correct document had come through the unnormalised path recovered it 94 percent of the time. Queries whose correct document had been bulk-loaded and normalised recovered it 44 percent of the time — the only way a norm-1 document reached the top ten was when few unnormalised documents happened to be even loosely on topic.

Those two figures reconstruct the aggregate exactly, which is what tells you the split is the whole story rather than part of it. The corpus divides sixty-two to thirty-eight between the two paths, so the overall recall is sixty-two hundredths of the 44 percent plus thirty-eight hundredths of the 94 percent. The first term is a little over a quarter, the second a little over a third, and they sum to 0.63 — the figure the evaluation job reported, with no residual left over for any other cause. Nothing else needs explaining, which is a stronger statement than "this is a plausible explanation".

What the similarity scores are telling you#

Score magnitude is the cheapest diagnostic available here, because under dot product the score inherits the document's norm directly. A result list whose scores span 0.8 to 11.4 is announcing a bimodal norm distribution, meaning the stored norms fall into two separated clusters rather than one continuous spread, and the check takes one query against the live index:

import numpy as np

# pull any 5,000 stored vectors back out of the index
vecs = np.array(index.fetch(ids=sample_ids)["vectors"])
norms = np.linalg.norm(vecs, axis=1)

print(f"min={norms.min():.3f}  median={np.median(norms):.3f}  max={norms.max():.3f}")
print(f"fraction within 1e-3 of 1.0: {np.mean(np.abs(norms - 1) < 1e-3):.3f}")

# min=1.000  median=1.000  max=14.087
# fraction within 1e-3 of 1.0: 0.618

Three of those four printed numbers are a signature rather than a measurement. A minimum of exactly 1.000 says some population was normalised, since nothing else produces that value to three decimal places. A median of exactly 1.000 says that population is more than half the corpus. A maximum of 14.087 says some other population was not, and by a wide margin. Only the fourth number is a quantity, and it is the share of the corpus in the first population.

That fourth number reads 0.618 against a corpus split of thirty-two thousand four hundred to nineteen thousand six hundred, which is a shade over sixty-two percent — and the small discrepancy is the sample rather than an error. Five thousand vectors drawn from fifty-two thousand estimate a proportion to within a bit under a percentage point, so 0.618 and the true share are the same finding. It is worth knowing that the printed figure is an estimate, because somebody will eventually try to reconcile it exactly against a row count and conclude that a third population exists.

A healthy normalised collection returns 1.000 for all three statistics and 1.000 for the fraction. A healthy unnormalised collection returns a smooth spread with a fraction near zero. Anything in between is a mixed population, and the fraction tells you how the corpus was split. This check costs one API call and rules in or out an entire class of migration failure, which is why it belongs near the top of the triage order rather than after a week of tuning.

Norm is also a rough proxy for content length, since a pooled representation of more text carries a larger magnitude, so the same query answers a second question at no extra cost: if the norms are smoothly distributed but far from 1, you learn nothing is normalised, and if they cluster in two bands you learn which subpopulation you are looking at.

The reason a defect like this survives is that the evaluation job reported one number. A single aggregate over 400 queries has no way to show that the failures are not spread evenly, and 0.63 is a plausible-looking figure for a retrieval system that is merely mediocre. The same 400 queries sliced by any attribute correlated with the defect make it obvious in one chart: recall by ingest path, recall by the date the gold document was written, recall by document length. Every one of those slices would have shown a clean bimodal split rather than a gradual curve, and the slice is a groupby over data the evaluation job already had. Retrieval evaluation that reports only an aggregate can tell you that something is wrong; it cannot tell you what kind of thing, and the shape of the failure is most of the diagnosis.

The defect is specific to metrics that read magnitude. Under cosine, a half-normalised collection is harmless — the index divides each stored vector by its own norm at scoring time, so a vector stored at norm 10.7 and the same vector stored at norm 1 produce identical scores and identical ranks. This is why the same drift can sit in a codebase for months without symptoms and then surface the day someone switches the index to inner product for the four percent latency saving, or the day a migration lands on a system whose configuration was copied from a -dot- model's documentation. Under Euclidean the drift bites in the opposite direction, pushing the unnormalised half away from every query and burying it rather than promoting it, which produces a corpus with a large permanently unreachable region and an evaluation curve that looks like a coverage problem.

Triaging the report, first two hypotheses#

The failure arrives as a sentence with no diagnostic content in it: same embeddings, same corpus, same queries, nothing touched in the model, recall@10 fell from 0.81 to 0.63. The sentence is doing more work than it appears to, because each clause eliminates something. Same model and same corpus rules out a content or encoder change; same query set rules out a shift in what users are asking. What is left is everything the migration touched, which is the storage and scoring layer, and that layer has a small number of ways to be wrong. Two candidate causes come from what a migration actually changes, and both are configuration rather than data.

The first is a metric mismatch, which the previous chapter covered: an old index that was IndexFlatIP contains no string saying "cosine" anywhere, so a port picks whichever metric the new system names most similarly, and the vectors are not the kind that makes those settings equivalent. The check is to read the index description and compare it against the model registry, which is one call and no data movement.

The second is the normalisation drift above, and its check is the norm histogram over a sample of stored vectors. These two interact: neither is a problem on its own if the other is in the right state. Normalisation drift with a cosine index is invisible, and a dot-product index over uniformly normalised vectors is correct. It takes both to produce the failure, which is exactly why each individual code review passed. A reviewer looking at the loader sees a normalize_embeddings=True that is correct on its own terms; a reviewer looking at the index creation sees a metric that is correct for the model card. Neither review has the other file open, and the defect exists only in the pair.

The size of the drop points at this class of cause rather than at anything gradual. Eighteen points of recall@10 on a 400-query set is not a handful of borderline queries slipping; it is a defect applying uniformly across the corpus, of the kind produced by a rule that treats one group of documents differently from another for a reason unrelated to their content. A cause that touched only the hardest queries, or only the tail of each result list, could not move an aggregate that far without also making the easy queries fail, and the easy queries here did not fail — the split by ingest path showed 0.94 recall on one half of the corpus, which is better than the old system's overall figure. A defect that leaves part of the corpus performing better than the system ever did is not a quality problem; it is a partition.

Repairing it without leaving the cause running#

Two things this diagnosis is not. It is not a data quality problem, and it will not be found by inspecting documents, because both halves of the corpus contain exactly the same text embedded by exactly the same model — the only difference is a division by a scalar. And it is not fixed by re-running the bulk loader over everything, or rather it is fixed by that only until the upsert worker writes its next document, at which point the corpus starts drifting again at whatever rate new content arrives.

That second point is the one that decides whether the incident recurs. A backfill treats the population that exists today and leaves the process that produced the split running, so the recall figure recovers, decays over the following weeks, and arrives back on the on-call rotation as a new incident with no obvious connection to the old one. The decay is also slow enough to look like ordinary drift rather than a returning defect, since it tracks the rate of new content rather than anything abrupt. Some systems let you enforce the invariant at the storage layer, which removes the possibility rather than repairing the consequence — pgvector's l2_normalize applied inside the insert statement, or a generated column that stores the normalised vector alongside the raw one and is the only column the index is built on. That is stronger than anything enforced in application code, because it survives a new ingest path written by someone who has never read this chapter.

There is also a mitigation that is not the fix, and it is worth naming because it is the right thing to do at 3am and the wrong thing to leave in place. Switching the index metric to cosine makes the symptom vanish immediately, since cosine divides each stored norm out and the two populations become indistinguishable — no re-embedding, no backfill, one configuration change and a re-index of vectors you already have. That is a legitimate incident response when the model's card documents cosine, in which case the metric was wrong anyway and you have fixed two things at once. It is the wrong resolution when the model documents dot product, because then you have traded a loud ranking failure for the quiet eleven-point degradation from the previous chapter, and the incident closes with a green dashboard and a defect intact. The two outcomes are told apart by the model card and by nothing that will be visible in the metrics afterwards, which is precisely why the decision has to be made before the dashboard turns green.

What makes the durable version durable is a write-time invariant rather than a write-time habit. A single embed_for_index() function that both paths call, which encodes, applies the registry's normalise setting, asserts the resulting norm against that setting, and is the only function in the codebase that returns a vector destined for this index, converts the failure from silent to impossible — the assertion fires in the worker's own logs the first time someone bypasses it. Pair it with a scheduled job running the norm histogram above against a sample of the live index each night and alerting when the fraction at norm 1.0 moves off its expected value, which catches the case the assertion cannot: a backfill script written outside the application, by someone who had a deadline and a CSV.

So a mixed-normalisation collection under dot product is not a subtle ranking degradation. It is a partition of the corpus into a population that can be retrieved and one that cannot, with the boundary drawn by which code path happened to write each row, and the arithmetic that produces it is one multiplication. Everything about diagnosing it is cheap — the score spread in a single result list, a norm histogram over a sample, a groupby on an evaluation set that already exists — and everything about it is invisible to an aggregate.

There is a third hypothesis for the same report, and it is the one most teams reach for first because it is the one with an obvious knob attached. The new index is approximate, the old one may not have been, and approximate means the search stops before it has seen everything.