A defensible model choice names domain, language, dimension and cost

An embedding model is a program you hand a piece of text and that hands back a fixed-length list of numbers, arranged so that texts about the same thing come back close together. Once you pick one, its output is written into a database and sits there for years as the coordinates every search is scored against, so the choice is expensive to reverse and is normally made in front of other people. That meeting is a design review: a proposal read by engineers who did not write it, who have the authority to decline it, and who have far less time than the author spent. What they are asking for is not a better model. They are asking for sentences somebody other than the author can check.

Defensible has a narrow meaning here, and it is worth separating from correct. A defensible justification names conditions a reader can test, so a reviewer can find out whether it is wrong without redoing the work; a correct choice can be defended badly, and a poor choice can be defended immaculately. The two properties are independent, and only one of them is available to somebody reading a document for the first time. A choice nobody can check is also a choice nobody can revisit.

The four things a reason has to name#

A justification that survives review states what the model has to do and shows evidence for each part. Four categories cover nearly everything a reviewer will ask, and the useful property of the list is that each item corresponds to a measurement someone has already made in the preceding chapters.

Organising a justification around categories at all rests on one step worth making explicit. A reason has exactly one job in a review, which is to eliminate something: a sentence that leaves every candidate exactly where it was has told the room about the state of the field rather than about this decision. For a sentence to eliminate, its answer has to differ — between the models on the shortlist, or between your documents and somebody else's. The four categories below were chosen against that test rather than for tidiness, and each of them returns a different answer for a Spanish-and-English support archive than it does for an English-only contracts archive.

Domain is the claim that the model handles your own documents and your users' query shapes. The evidence is the head-to-head on your golden set — a fixed query set with recorded judgments about which chunks answer each query, run against both candidates (chapter 3, where the corpus snapshot, the chunker, the query set, the judgments, the cutoff and the index parameters are all held still and only the encoder moves). It is quoted with the number, the sample size, the confidence interval, and the per-stratum breakdown that shows the win is not concentrated in one corner of the traffic. Language is the claim that every language you serve is covered, evidenced by the per-language stratum rather than by the model card's list of supported languages, because a model that lists 94 languages is often strong in eight of them.

Dimension is the claim that the vectors fit your storage and latency budget, evidenced by the arithmetic at projected corpus size rather than current size — the storage arithmetic (chapter 4, where the number of vectors, times the dimension, times the bytes per component gives the raw figure, which the index structure and the replicas then multiply). Cost covers both the recurring bill and the operational exposure: hosted or self-hosted, snapshot pinned or not, what detects drift, and what the next migration would cost if you had to run it. Those last two items are the hosting trade-off (chapter 6, where a provider silently updating weights leaves an index holding two vintages of vectors that no longer compare) and the migration bill (chapter 5, where changing the model means re-embedding and reinserting every stored chunk).

Four sentences, one per category, each carrying a number. That is the whole artefact, and it is short enough that its brevity is sometimes mistaken for a lack of rigour by people who have seen twenty-page evaluation documents that contain no numbers about the corpus in question.

The four categories are also a sequence, and running them in order saves work for a reason that is worth multiplying out. Dimension and cost are the cheapest to evaluate, because they need arithmetic and a model card rather than an experiment, and they eliminate candidates before anyone spends judging time on them: a 3072-dimension model that would cost $5,400 a month resident at your projected corpus size is out unless it wins by an amount that justifies the bill, and knowing that in advance sets the bar the experiment has to clear. Language coverage is next, since a model that does not handle one of your two languages is out regardless of its English scores. Domain evidence comes last because it is the expensive one, and by then it is being gathered on two or three survivors rather than eleven candidates.

The saving from that order is the per-candidate cost of the expensive step, multiplied by the number of candidates the cheap steps removed. Judging is the expensive step and it is roughly fixed per candidate once the query set exists, so eliminating eight of eleven candidates on arithmetic saves eight times that figure — most of an afternoon each — and the arithmetic itself costs one multiplication per candidate. Running the order the other way does not produce a different answer, it produces the same answer at several times the price, and it produces it late enough that the sunk judging effort starts arguing for the candidate it was spent on.

A criterion that could have come out the other way#

The tempting mistake in this whole area is not an arithmetic slip. It is a belief that a checklist is a defence, and it survives because a long list of criteria looks more rigorous than a short one. A proposal that names accuracy, maturity, community support, production-readiness and ecosystem fit reads as thorough, and every one of those five terms describes something real. The question a reviewer should ask is what each one eliminated.

Follow the step through. A criterion earns its place by partitioning the candidates into ones that pass and ones that fail. Partitioning requires the criterion to return different answers for different inputs. A criterion such as "the model must produce high-quality embeddings" returns yes for every serious candidate on any corpus anybody would build, so it partitions nothing, so it eliminates nobody, so its presence in the document changes no decision. It is not a criterion; it is a formality wearing the clothes of one. The strong form of the test is to hold the criterion still and vary the corpus: if the answer does not move when you swap a Spanish-and-English support archive for an English-only contracts archive, the criterion was never discriminating between corpora, and a selection checklist that is constant when the corpus varies has documented nothing about this corpus.

Set the four categories against that test and they behave differently. Language returns yes for a model on an English-only archive and no for the same model on one where a quarter of the tickets are Spanish. Dimension returns yes at two hundred thousand chunks and no at ten million for the same 3072-dimension model, because the same width is a rounding error at one scale and a five-figure annual bill at the other. Domain returns whichever way the golden set says, and the whole point of running it is that nobody can predict which. Cost inverts on whether a dated snapshot is on offer. Each of the four is a question the corpus answers, which is what makes an answer to it evidence.

Stating the bar before the experiment is the same property applied to the threshold, and it is what turns the eventual number into a decision rather than a data point. "We will migrate if the challenger's confidence interval clears +3 points of recall@5 and the Spanish stratum does not regress" is a sentence that can be written before any embedding is computed, and it removes the argument that otherwise happens when the result lands at +2.4 with an interval that includes both zero and five. A bar chosen after the number is visible can be placed on either side of it, so it returns yes whatever the experiment found — which is exactly a check that is constant when the thing it is meant to discriminate varies. Reviewers notice when the threshold appears after the measurement, and they are right to.

Which claims survive one follow-up question#

Reviewers do not have time to audit a study, so a good review works by asking each claim the single question most likely to expose it. Claims that collapse tend to collapse on the first question, and the pattern is consistent enough to anticipate.

Read the third column for where the rows are far apart rather than for how each one scored. The four collapsing rows and the four surviving ones are not spread along a scale of quality: they sit at two ends of a single property, which is whether the evidence existed before the review or was promised for after it. The prefix row makes that visible, because the claim "it scored better in our test" is about this system and still collapses — the follow-up asks whether each candidate was given the short instruction string its own model card documents on the query side and on the document side (chapter 2), and a harness that hardcoded one string for both arms answered the question before the experiment ran. The rows that collapse are all claims about the world in general offered in place of a claim about this system.

The row that behaves differently is the last one, and its "partially" is the interesting value in the column. A latency claim measured single-threaded is not false and it is not evidence for the thing it is offered as evidence for, since p95 under concurrent load — the figure only one query in twenty exceeds — is where a wider vector starts competing for memory bandwidth with every other query in flight. That row is the shape to watch for generally: a real measurement of a neighbouring quantity, offered in place of the one that matters.

One claim is worth pre-empting rather than answering, and it is the one the table does not contain. A reviewer who asks "what does the incumbent plus a reranker score" — a reranker being a slower second model that reads a query and a candidate document together and rescores them, changing no stored vector — and receives an answer has just been told the proposal considered the cheap alternative and measured it. A reviewer who asks and receives a promise to check has been told the opposite. The general form is that every proposal to change a stored artefact should carry the best non-stored alternative alongside it, measured on the same set, and let the margin between them carry the argument for the migration.

Recording the reasoning rather than the conclusion#

Whatever gets written down is what the next engineer inherits, and most teams write down the conclusion. The usual vehicle for writing down the rest is an architecture decision record, or ADR: a short numbered file kept in the repository that states one decision, the evidence behind it, the options rejected, and the conditions under which it should be revisited. A configuration file containing EMBEDDING_MODEL=text-embedding-3-small transmits the decision and none of the reasoning, so when a new candidate appears eleven months later, whoever evaluates it starts from nothing: no golden set, no record of what was rejected or why, no baseline number to beat, and no knowledge that the 512-token limit of one obvious alternative was the reason it lost. That work is two or three weeks the first time and an afternoon when the record exists.

# docs/decisions/0031-embedding-model.md (front matter)
decision: "Use voyage-3-lite@2025-02-14 as the retrieval encoder"
date: 2025-06-11
owner: search-platform
status: accepted

evidence:
  golden_set: "eval/golden/2025-05.jsonl, 120 queries, pooled judgments over top-20 union"
  harness_commit: "a1c93f7"
  primary_metric: "recall@5"
  result: "0.812 vs 0.766 for incumbent bge-base-en-v1.5, delta +4.6"
  interval: "95% paired bootstrap [+1.5, +7.7]; sign test 32/47 non-tied, p=0.02"
  strata:
    english_keyword: "+3.1 (n=61)"
    english_question: "+5.9 (n=34)"
    spanish: "+6.2 (n=25)"
  held_constant: "corpus snapshot 2025-05-30, 480-token chunks, HNSW m=16 ef_search=64, cosine"
  prefixes: "each candidate used its own documented strings; see harness encoders.py"

shape:
  dimension: 512
  projected_chunks: 10_000_000
  raw_vectors: "20.5 GB"
  resident_with_index_and_3_replicas: "67 GB, ~$920/month"
  max_sequence: 32000

hosting:
  mode: hosted
  pinned_snapshot: "voyage-3-lite@2025-02-14"
  fingerprint_column: "chunks.embedded_by"
  drift_detection: "canary_256 daily, alert < 0.999; 20 rank canaries hourly"
  reindex_cost: "16.8B tokens, ~$1,000, ~40h wall clock at current rate limit"

rejected:
  - model: "a 3072-dim frontier model"
    reason: "won by 1.9 recall@5, inside the interval; 123 GB resident, ~$5,400/month"
  - model: "an open-weight 7B instruction encoder"
    reason: "+5.1 recall@5 but 2×A10G always-on for query path, ~$1,500/month, and 90ms p95"
  - model: "incumbent + cross-encoder reranker"
    reason: "+3.8 recall@5 for no reindex; kept as a follow-up, not an alternative"

revisit_when:
  - "Spanish share of traffic exceeds 40% (currently 21%)"
  - "chunk size changes, which invalidates the judgments"
  - "corpus passes 25M chunks, which re-opens the dimension arithmetic"
  - "provider announces deprecation of the pinned snapshot"
  - "annually, by re-running eval/golden against the current top three"

The first thing to do with a record like that is check it against itself, because a record whose own numbers do not reconcile is a record nobody verified. Start with the strata block. Add the three query counts — sixty-one keyword, thirty-four question, twenty-five Spanish — and the sum is exactly the 120 queries the golden_set line names. That closure is worth more than it looks: because there is no residual, there is no fourth group of queries sitting outside the breakdown, and the headline delta therefore cannot be hiding a subgroup that moved the other way. A stratum table whose counts fall short of the query total is one where the missing rows are precisely the ones nobody wanted to show.

The strata reconstruct the headline as well as the count, and the operation that does it is a weighted mean. Multiply each stratum's delta by the number of queries in it, add the three products, and divide by 120. Weighting by count is the right operation because the headline is an average over queries rather than over strata: each query contributes one score, so a stratum with more queries in it pulls the mean harder, and treating the three strata as equals would let twenty-five Spanish queries outvote sixty-one keyword ones. Worked through, the three weighted contributions land a hair under the +4.6 the result line states, and the small shortfall is what one-decimal rounding on three strata plus three-decimal rounding on the two recall figures can produce. Nothing is missing from the decomposition; the residual is rounding, and knowing that is the difference between a total you trust and a total you hope about.

Move one weight and the ranking of the reasons moves with it. Spanish carries the largest per-query gain of the three strata, so if Spanish traffic grew the weighted mean would climb toward it, and if the keyword stratum grew the mean would fall toward the +3.1 that stratum contributes. That sensitivity is the whole content of the first revisit trigger: "Spanish share of traffic exceeds 40% (currently 21%)" is not a general instruction to keep an eye on things, it names the weight whose movement changes the arithmetic that justified the decision.

The shape block reconciles the same way, against the storage arithmetic rather than the evaluation. Five hundred and twelve dimensions, times ten million chunks, times four bytes for each float32 component, is the 20.5 GB on the raw_vectors line — the same multiplication the third row of the table above puts to a proposer who has not done it. Multiply that raw figure by the three replicas and the product is a little over sixty gigabytes, which leaves a few gigabytes between it and the 67 GB stated. That gap is the graph structure the vector database builds to find near neighbours without scanning everything, and it is charged per vector rather than per float, so it stays roughly the same whatever the dimension. The residual has a name and a cause, which is what lets you treat 67 GB as complete rather than as a figure with unexplained slack in it.

That reconciliation is also what makes the rejected block readable, and it repays a careful look because its storage figure is quoted on a different footing from the accepted one. Three thousand and seventy-two dimensions, times the same ten million chunks, times the same four bytes, comes to the 123 GB in the first rejected entry — which is the raw vector figure, arrived at by the multiplication that produced 20.5 GB for the accepted model, and not the resident figure that 67 GB is. Carried through the same replica and graph multipliers, the resident total for that candidate would be several times what the entry prints, and the monthly figure beside it is consistent with that larger number rather than with 123 GB. A number in a record has to say which quantity it is: raw and resident differ here by more than a factor of three, and a reader comparing 123 against 67 will conclude the frontier model costs roughly twice as much when the bill says nearly six times.

Read the quality column of the rejected block as gaps rather than as levels and the decision becomes legible. Every delta in it was measured on the same 120 queries with the same instrument, and that instrument resolves about ±3.1 points — the paired bootstrap interval on a set that size. The accepted model's +4.6 over the incumbent carries an interval of [+1.5, +7.7], which excludes zero, so that one gap is readable. The frontier model's 1.9 sits inside the interval and is smaller than the instrument's resolution, so no amount of staring at it will tell you whether that model is better. The open-weight 7B's +5.1 and the reranker's +3.8 straddle the accepted +4.6 by less than the resolution in both directions, which means the golden set cannot separate any of those three on quality at all.

That is why the cost column decides. Where the quality gaps are unreadable, the gaps that are far apart are the ones in dollars and milliseconds: about $920 a month against about $5,400 against about $1,500 of always-on GPU, and a 90ms p95 on the query path for the open-weight option. Ranking these candidates by their recall deltas would be sorting noise; sorting on the column that separates them by more than the instrument's resolution is the only axis carrying signal. The reranker entry is filed as a follow-up rather than as a defeated alternative for the same reason — its gap to the accepted model is inside what the set can resolve, so it was deferred rather than beaten.

The rejected block therefore does more work than the accepted decision. It records that a much larger model was tried and won by an amount inside the confidence interval, which stops the next person from proposing it as an obvious upgrade. It records the reranker result, which belongs on a backlog rather than in a comparison. And it records the always-on cost that ruled out the open-weight option, which is the number that changes if query volume grows by an order of magnitude — making the record self-invalidating in a useful way.

Where the record lives determines whether it is read. A decision document in a wiki is found by people who already know it exists, which excludes exactly the engineer who will change the model configuration eleven months from now while chasing something else. Keeping it in the repository next to the code, numbered, and referenced by a comment at the configuration site — a single line saying which document justifies this value — puts it in front of the person editing the value at the moment they edit it. The same applies to the golden set and the harness: a decision record that points at an evaluation nobody can run is a historical document, and one that points at eval/golden/2025-05.jsonl and a make target is an instrument.

The record should be checkable rather than aspirational, and the cheapest way to keep it honest is an assertion that runs in continuous integration and fails when the deployed configuration and the document disagree.

import yaml

RECORD = yaml.safe_load(open("docs/decisions/0031-embedding-model.md").read().split("---")[1])

def test_deployment_matches_decision(settings):
    assert settings.embedding_model == RECORD["hosting"]["pinned_snapshot"], (
        "Deployed encoder differs from ADR-0031. A change here rewrites every "
        "stored vector's meaning; update the decision record and plan a migration."
    )
    assert settings.embedding_dimension == RECORD["shape"]["dimension"]
    assert settings.query_prefix == EXPECTED_PREFIXES[settings.embedding_model]["query"]
    assert settings.doc_prefix == EXPECTED_PREFIXES[settings.embedding_model]["document"]

That test catches the two changes most likely to be made casually: someone switching a pinned snapshot to a floating alias because the pinned one produced a deprecation warning, and someone editing a prefix while debugging and leaving it. Both are one-line diffs that pass code review and both silently split the index into two vintages. The assertion is worth more than a paragraph asking engineers to keep the document current for the property this whole chapter turns on: the test can fail. A request in a wiki returns the same result whether or not the configuration has drifted, which makes it a formality; an assertion that compares the record against the deployed settings returns a different result in the two cases, which makes it a check.

A decision record does not make a choice correct, and a team can produce an immaculate document justifying a model that a better experiment would have rejected. Its value is that it makes the choice auditable and cheap to revisit, which are different properties from being right. When the record is wrong, the record is also what makes the wrongness discoverable, since every claim in it is a number someone can re-measure.

How the numbers go stale#

There is a specific way these documents rot, and it is that the numbers stay while the conditions underneath them move. A record stating a footprint of 67 GB and $920 a month was true against a projection of ten million chunks, and if the corpus reaches twenty-five million the sentence is still in the document, still precise, and no longer describing anything. The failure is quiet: the number goes on looking exactly as authoritative as it did on the day it was computed. Dating every quantitative claim and naming the input it was computed from — "at 10M chunks, projected 2025-06" rather than "67 GB" — makes the staleness visible to a reader instead of requiring them to reconstruct it.

The revisit triggers do the same job from the other direction by naming the input values whose change should send someone back to the arithmetic, and the useful discipline is to write one trigger per input the record multiplied by. The shape block multiplied a chunk count by a dimension; the chunk count has a trigger at twenty-five million and the dimension is fixed by the model, which is why that block needs one trigger rather than two. The evidence block was weighted by traffic mix, which is why the Spanish share gets a trigger of its own. A trigger list assembled by asking "what would worry me" produces vague entries that never fire; one assembled by listing the inputs to each calculation produces entries that name a number and a value it must not cross.

It is also not a freeze. The revisit triggers exist because the conditions the decision rested on are the sort that change: traffic mix moves as a product expands into new markets, corpora grow by an order of magnitude, chunking gets revised for reasons that have nothing to do with the encoder, and the field itself moves fast enough that a two-year-old choice deserves an afternoon of re-testing regardless of whether anything else changed. A team that treats the record as settled law ends up defending a model on the strength of a measurement taken against a corpus that no longer exists.

The single trigger worth watching most closely is the chunking one, because it is the one that arrives disguised as unrelated work. A change from 480-token chunks to 800-token chunks is usually proposed as a retrieval-quality improvement and evaluated on its own terms, and it invalidates the judgments that justified the model, requires a full reindex anyway, and may well change which model wins. The mechanism is worth stating rather than asserting: pooled judgments record that a particular chunk answers a particular query, rechunking destroys those chunks and creates different ones, and a judgment about an object that no longer exists cannot be scored against anything. When those two changes land in the same quarter, they should be evaluated together, with the chunking change made first and the model comparison re-run on the new chunks.

A defensible model choice is therefore not a longer document than an indefensible one, and it is often shorter. It is four sentences, one each about whether the model handles your material, whether it handles your languages, whether its vectors fit the budget you will have rather than the one you have now, and what it costs to run and to leave — with a number behind each and a condition under which each would have read the other way. What makes it hold up in a room is that every claim in it was produced before somebody asked, and what makes it survive the year after the room is that it is stored where the next person will trip over it, with the inputs to its arithmetic named so that a reader can see when they have moved.

What none of this settles is how the chosen vectors get used once they exist: which store holds them, how the index parameters are tuned against the recall and latency budget the record commits to, how filters and hybrid lexical arms combine with dense retrieval, and how the whole path is monitored in production. The decision record is the input to that work — it hands over the model, its fingerprint, its prefixes, its dimension, the footprint it implies and the recall it was measured at, which is everything the retrieval layer needs to be built against.