A new encoder invalidates every vector you have already stored

The final evaluation came back at +3.1 nDCG@10 against the frozen baseline: 0.58 to 0.61 over the same 240 judged queries, the same corpus snapshot, the same k, the same judgments, run through the same harness as the baseline six weeks earlier. It is a real gain measured the way stage 8.3 says to measure it. The migration that puts it in front of users is 84 million chunks re-encoded, and the estimate on the whiteboard was six weeks and about $40,000.

That number is worth breaking down, because the intuition that re-embedding is a GPU bill is wrong and leads people to under-price it by an order of magnitude. Encoding 84 million chunks of roughly 250 tokens with a 768-dimensional base model runs at about 900 chunks per second on one A10G, so 26 GPU-hours, or under $100 spread across a few cards. The money is elsewhere: three engineers for six weeks at fully-loaded cost is around $34,000; running a second index alongside the live one for the duration adds 84M × 768 × 4 bytes = 258 GB of vectors, which on a managed vector store at $25 per GB-month is $6,400 a month; and the re-ingest re-runs the whole document pipeline, which surfaces the 3% of documents that fail to parse and someone has to look at each of them. The compute is a rounding error against the people and the doubled infrastructure.

Every step of that breakdown closes, and the closure is what makes the estimate usable rather than a guess with a large number in it. Eighty-four million chunks at nine hundred a second is a little over ninety thousand seconds, which is the 26 GPU-hours quoted. Three engineers for six weeks is eighteen engineer-weeks, which is the figure the decision record below carries. Eighty-four million vectors of seven hundred and sixty-eight dimensions at four bytes a dimension is the 258 GB quoted, and 258 GB at twenty-five dollars a gigabyte-month is the $6,400. Add the people, the doubled storage and the compute and the total lands on the whiteboard's $40,000 with the GPU hours contributing a fraction of a percent of it. So the item that dominates everybody's mental model contributes almost nothing, and the two items that dominate the bill — salaried time and a second copy of the index — are the ones nobody puts in a slide about embedding costs.

Read the evaluation by query class before reading the total#

An aggregate over 240 queries is an average over populations that were not affected equally, and a domain fine-tune produces exactly the pattern that averaging conceals: large gains on the queries whose language was in the training data, and losses on queries that were already working and were not represented.

The classes have to be defined before the experiment and have to be large enough to read. Stage 8.3's golden set was already stratified, so the strata are usually the classes, and where they are not the taxonomy comes from the query log: cluster a few thousand real queries, name the clusters by hand, and keep any class with at least 25 judged queries as a reporting row, folding the rest into an "other" bucket rather than reporting a delta over six queries. Language, query length band, and whether the query contains an identifier are three splits worth having in every taxonomy because each corresponds to a way training data can be unrepresentative. Defining them afterwards, once the aggregate is in hand, invites the same search for a favourable slice that the pre-registration below exists to prevent — so write the class list into the decision record alongside the thresholds.

Two things about that table close exactly and one does not, and the pattern is the reason to read a table like this by its gaps rather than by its levels. The five class counts add to 240 with nothing left over, so every judged query sits in exactly one reporting row and the "all queries" line really is an average over the same population the classes describe — there is no unclassified remainder hiding a sixth behaviour. The Δ column also closes: weight each class delta by its own count, add them, divide by 240, and the result is the +0.03 on the bottom row. What does not close is the levels. Weighting the baseline column the same way lands a hundredth above the 0.58 printed beneath it, and weighting the fine-tuned column lands a hundredth above the 0.61, because each cell was rounded to two places before it was printed. The rounding error survives in the levels and cancels in the difference, since it is very nearly the same error in both columns and subtraction removes what is common to them. That is why the headline delta can be trusted from this table while neither headline level can be reconstructed from it exactly.

The German regression is the one that would have shipped invisibly. Twenty-six queries out of 240 cannot move an average by much, and the training set — mined from click logs and support tickets generated overwhelmingly by English-speaking users — contained almost no German, so three epochs of contrastive training on English pairs drifted the multilingual base model's German region without any signal defending it. That is ordinary catastrophic forgetting, the name for a model losing competence it already had in regions that no training example defended, and the standard mitigations are to mix in a replay set of general-domain pairs at 10–20% of the batch, to train fewer epochs at a lower learning rate, or to freeze most of the encoder and adapt with LoRA — low-rank adaptation, which trains a small pair of added matrices and leaves the original weights untouched — so the base weights move less. Each of those trades some of the in-domain gain for stability, and choosing among them requires the per-class table to exist in the first place.

Why a class with no training examples moves at all is the step that makes forgetting predictable rather than mysterious, and it follows from what chapter 4 established. A fine-tune changes the encoder's parameters, and every document in the corpus is placed by those parameters, so a class that contributed no training row is still re-placed by the training other rows caused. Gradient descent has no term that preserves behaviour it was not asked about; a region of the space with no example defending it moves wherever the shared weights carry it. The mitigations follow directly from that sentence rather than from experience: a replay set puts examples back into the region so that something defends it, a lower learning rate and fewer epochs shorten the distance the weights travel, and LoRA constrains what can move at all. Three remedies, one mechanism.

The short-navigational regression has a different cause and the same lesson. Those queries were already at 0.71, the highest baseline in the table, so there was nothing for the training to add and plenty for it to disturb, and the pairs that would have protected them — two-word queries to a landing page — were filtered out by chapter 2's length(s.query_text) BETWEEN 8 AND 300 clause. The filter that improved training-set quality removed an entire class of query from the model's supervision.

That is a causal chain running the whole length of the topic and it is worth naming as one, because it is the kind of thing no single stage's review would catch. A filter written in chapter 2 to raise the precision of mined pairs excluded short queries. Short queries are the class that was already working best. A class with no pairs has nothing defending it during training. Therefore the class that was working best is the one the fine-tune degrades. Each link is checkable in the artefact of its own chapter, and no reviewer looking at the SQL, the training config or the evaluation on its own would see the chain.

Three points on 240 queries may not be three points#

Before any of this becomes an argument, the aggregate needs an interval around it. A paired bootstrap over per-query deltas is a dozen lines and settles whether the number is a result. The procedure resamples your own per-query differences, with replacement, a few thousand times and reports the range the average falls in, which is how far the headline number could move if you had judged a different sample of queries.

import numpy as np

def paired_bootstrap(base_scores, new_scores, n=10000, seed=0):
    """base_scores, new_scores: per-query nDCG@10, aligned by query id."""
    d = np.asarray(new_scores) - np.asarray(base_scores)
    rng = np.random.default_rng(seed)
    idx = rng.integers(0, len(d), size=(n, len(d)))
    means = d[idx].mean(axis=1)
    return {"delta": float(d.mean()),
            "ci95": (float(np.percentile(means, 2.5)),
                     float(np.percentile(means, 97.5))),
            "p_worse": float((means <= 0).mean()),
            "n_improved": int((d > 0).sum()),
            "n_regressed": int((d < 0).sum())}

Pairing is what makes a dozen lines enough, and it is the design decision inside that function. The quantity resampled is the per-query difference between the two systems, not the two systems' scores separately. Every query has its own intrinsic difficulty, and that difficulty appears in both scores, so subtracting removes it before any resampling happens. What is left varies only with how the two systems differ on that query. An unpaired comparison would have to see through the query-to-query variation in difficulty, which is far larger than the effect being measured, and would need many more judged queries to do it.

On this run it returned a delta of +0.031 with a 95% interval of [+0.004, +0.058], 118 queries improved, 79 regressed, 43 unchanged. The interval excludes zero, so the gain is real in the statistical sense, and its lower bound is +0.004, which is indistinguishable from nothing. A result that is somewhere between negligible and moderate is not a result that pays for a six-week migration, and reporting only the point estimate is what makes it look like one.

Two arithmetic facts in that output are worth checking, because each licenses a reading. The three query counts add to 240 exactly, so every judged query is accounted for and none was silently dropped by a join — which also says that nearly two queries in five of those that moved at all moved the wrong way, a fact the aggregate delta cannot express. And the interval is symmetric about the delta: the distance from +0.004 up to +0.031 is the distance from +0.031 up to +0.058. A symmetric bootstrap interval means the resampled distribution is close to a bell, which licenses reading its half-width as an ordinary margin of error and comparing it against the effect directly. Here the half-width is very nearly as large as the effect, which is the whole finding in one sentence.

This is where the chapter's stumble lives, and it is a belief that a defence works rather than a slip in the arithmetic. A confidence interval that excludes zero is the standard evidence that a result is real, and it is real: repeat the experiment on a different sample of queries and the sign would very probably come back the same. What the test cannot do is say whether the effect is worth $40,000, because the same interval-excludes-zero verdict holds for an effect of four thousandths and for an effect of half a point. The verdict is constant across the entire range of the quantity the decision turns on. Statistical significance answers "is there an effect", the migration is priced against "how big is it", and a reader who has satisfied the first has not begun on the second. The lower bound is the number to quote, because it is the version of the result that is compatible with the evidence and least flattering to the person who ran it.

Name the bar before you run the experiment#

The failure mode this chapter exists to prevent is not a wrong calculation. It is that the person doing the arithmetic proposed the project, defended it in three meetings, and has spent six weeks on it, so every ambiguous number resolves in the direction of shipping. The defence against that is to write down, before training starts, what result would justify the migration, and to make the criteria specific enough that they cannot be reinterpreted afterwards.

# decision-record.yaml — committed before the first training run
experiment: domain-finetune-encoder-v1
baseline: bge-base-v1.5, index snapshot 2026-03-04, judgments v7, k=10
migration_cost: {engineer_weeks: 18, dollars: 40000, calendar_weeks: 6}

ship_if:
  - metric: ndcg@10
    delta: ">= +0.06"
    ci95_lower: "> +0.03"
  - metric: recall@50
    delta: ">= +0.05"
  - condition: "no query class of n>=25 regresses by more than 0.02"
  - condition: "gain survives with the reranker enabled"

do_not_ship_if:
  - "gain is smaller than the reranker-only gain already in production"
  - "a re-index is not otherwise planned within 2 quarters"

alternatives_if_no:
  - "keep the pairs, the mining pipeline and the eval harness"
  - "retrain the cross-encoder on the refreshed negatives instead"
  - "revisit when the planned 2026-Q4 chunking change forces a re-index anyway"

Measured against that record the run fails on three of the four ship conditions, which is a much easier conversation than an open-ended argument about whether three points feels like enough.

Which three is worth checking off against the numbers above rather than taking on trust, because the point of a record like this is that anybody can do the check. The nDCG condition asks for a delta of at least +0.06 and a lower bound above +0.03, and the run produced +0.031 with a lower bound of +0.004, so it fails on both of its clauses. The class condition forbids any class of at least 25 judged queries regressing by more than 0.02, and two classes do — the German subset at −0.07 over 26 queries and the short navigational at −0.04 over 38 — so it fails as well. The reranker condition is the third, and it is the one the paragraph below is about. The recall@50 condition is the only one this chapter's own evidence does not adjudicate, which is itself a reason to have written it down: an unmeasured condition is visible as unmeasured, where an unwritten one is simply absent.

The fourth condition — that the gain survives with the reranker enabled — is the one teams forget and the one that most often kills the project, because the fine-tuned encoder and the fine-tuned reranker learned overlapping things from the same pairs, and a +3 measured with reranking switched off can be +0.8 with it switched on.

The last line of the record is the move that turns a no into a deferral with a date. Re-embedding is enormously cheaper when it rides along with a re-index you were going to do anyway — a chunking strategy change, a dimension reduction for cost, a base-model upgrade — because the engineering weeks and the doubled index are already budgeted and the marginal cost of also swapping the encoder is close to the 26 GPU-hours. Keeping the checkpoint, the evaluation and the decision record on a shelf until that window opens is a legitimate outcome that captures most of the value at a fraction of the price.

If the answer is yes, the migration has one hard rule#

When the numbers do clear the bar, the engineering has a single invariant: no query is ever answered from an index containing vectors from two encoders. Mixed vectors do not error, they return plausible nonsense, because a cosine between an old vector and a new query vector is a well-formed float with no meaning, and the failure presents as a gradual quality complaint rather than an alert.

That absence of an alarm is worth one more sentence, because it is the same shape as everything else in this chapter. A cosine similarity is defined for any two vectors of equal length; nothing in the arithmetic knows which model produced either of them, and there is no dimension, magnitude or range check that a stale vector would fail. So the health signal stays green across the entire range of the dangerous property, from an index that is one per cent stale to one that is half stale, and the only instrument that moves is user complaints weeks later. An invariant is the right response to a failure with no detector: make the state impossible rather than watching for it.

The safe shape is a second index built alongside the live one, written by a backfill job that reads the same chunk store, with reads switched over atomically by a config flag once the backfill is verified complete. Verification means comparing counts and checksums per shard rather than trusting the job's own log, since a re-ingest at this scale will drop documents that fail to parse and those are exactly the ones nobody notices missing.

Order the backfill by traffic so the useful part lands first: the 2% of documents that receive 60% of clicks can be encoded in an hour, which lets a shadow evaluation run against real queries days before the full corpus finishes. New and updated documents during the migration window have to be written to both indexes, so the ingestion path needs a dual-write for the duration and a switch to remove it afterwards; forgetting the removal leaves a job writing to a deleted index, which fails loudly and is the least dangerous of the available mistakes. Keep the old index for at least a week after cutover, because that is the rollback, and the alternative rollback is a second six-week re-embedding.

Offline numbers should also be confirmed online before the old index is deleted. Interleaving is the cheapest instrument: for a fraction of traffic, merge the two rankings into one list and attribute each click to the model that contributed the result, which controls for position bias in a way that a straight A/B on click-through rate does not. A week of interleaving over a few thousand queries usually resolves a three-point offline difference more decisively than the 240-query judged set did, and it measures the thing the judged set only approximates.

The inventory a no leaves behind#

Deciding not to ship the encoder does not undo the work, and the inventory is worth stating because it is the reason this is a normal outcome rather than a failed project. The 20,000 pairs remain, and they are the scarce asset: they trained the reranker that is already in production and they will train whatever encoder replaces this one. The mining pipeline remains, and it is the piece that took longest to get right. The evaluation harness now reports per-class deltas with confidence intervals, which will make every subsequent retrieval decision faster. The triage from chapter 1 fixed an ingestion bug and a chunking bug that were costing recall regardless of any model.

There is also an honest limitation that argues for restraint independently of the arithmetic. A fine-tuned encoder is frozen against a vocabulary that keeps moving: next quarter's part numbers, the product line acquired in June, the shorthand a new support team invents. Each refresh repays the migration cost, so adaptation is not a one-time expenditure but a recurring one with a cadence set by how fast your domain drifts. A reranker refresh, by contrast, is a model file swap with no data migration, which is why volatile domains tend to converge on a modest base encoder with hybrid retrieval and a frequently-retrained reranker in front of it. Stable domains with vocabulary the encoder genuinely cannot represent — clinical shorthand, a low-resource language, a chemical nomenclature — are where the encoder work pays back over years and where the ceiling cannot be raised any other way.

The two costs behave differently under repetition, which is the part that decides architecture rather than a single release. The encoder's price is paid per migration and the migration is dominated by engineer time and a doubled index, so a domain that drifts twice a year pays it twice a year whatever the corpus size. The reranker's price is paid per query, so it scales with traffic and not with drift. Those are different denominators, and matching the one that grows fastest in your organisation to the model whose cost sits in it is most of the long-run decision.

The loop this topic describes is short enough to run in a fortnight and is meant to be run rather than read: diagnose with the numbers you already have, pull pairs from the logs before the retention policy deletes them, generate what the logs do not cover and check the overlap, train with a controlled batch and mined negatives you have filtered, train the reranker on the same file, and measure both against a baseline you froze before you started. The Sentence Transformers training documentation and Hugging Face's fine-tuning guide are the reference for the mechanics; the parts that decide the outcome are the data assembly and the decision record, and both of those are yours. Start with the triage in chapter 1 on your own evaluation output, because it costs an afternoon and it is the step that tells you whether any of the rest applies to you.