A threshold is a property of the model that produced the scores

A retrieval system returns a ranked list, and something downstream has to decide where to stop reading it. The usual instrument is a constant: keep every result scoring above some number, discard the rest. That constant is almost always inherited — chosen once by somebody looking at a few dozen results, then carried through refactors, model upgrades and chunking changes as though it described the task rather than the instrument that produced the scores.

It describes the instrument. A similarity score is the output of one model compared under one metric, and two models put their relevant and irrelevant pairs in entirely different places. What matters for a cutoff is not where those places are but how far apart they sit relative to how wide each group spreads, which is a quantity nobody can read off a single score and which no retrieval benchmark reports. The rest of this chapter is the measurement that produces it, why the obvious repair after a model swap provably does nothing, and the instruments that survive the swap.

Two models, two different meanings for a high score#

The only way to know what a score means for a given model is to measure it over pairs whose relevance you already know. Collect a few hundred query-document pairs that are genuinely relevant and a few hundred that are not — the negatives can be sampled from the same corpus, or better, drawn from the top-50 results for other queries so they are plausible rather than random — then score them and look at the two distributions.

Here is what that measurement returned for two candidate models on the same 600 labelled pairs:

Only the first four rows of that table were measured. Everything below them is derived from those four numbers, and working the derivations shows what each summary is actually claiming.

The gap row is subtraction: 0.847 less 0.781 is 0.066, and 0.702 less 0.253 is 0.449. d′ in that table is the difference in class means divided by the pooled standard deviation, borrowed from signal detection, and it is a more useful summary than the gap alone because the gap is meaningless without the spread. Model B's classes are 0.449 apart with standard deviations near 0.12; Model A's are 0.066 apart with standard deviations near 0.04. Dividing each gap by its own spread gives the 3.73 and the 1.76, and the ordering survives the division — B is genuinely better separated — but the ratio between them is nothing like the ratio between the raw gaps, which is the whole reason to divide.

The last two rows come from the same four parameters plus one assumption: treat each class as roughly normally distributed around its mean. Under that assumption a cutoff's effect is decided entirely by how many standard deviations it sits from each mean. For Model A, 0.75 lies a little under three quarters of a standard deviation below the negative mean of 0.781, and a normal distribution puts about three quarters of its mass above such a point — which is the 76% of negatives admitted. The same 0.75 lies a little over three standard deviations below Model A's positive mean, and almost nothing sits that far into a tail, which is the 99.9% of positives kept. For Model B, 0.75 is nearly four standard deviations above the negative mean of 0.253, so essentially no negative reaches it, and a little under half a standard deviation above the positive mean of 0.702, so about a third of positives clear it. Four cells, four substitutions, no residual.

The AUC row can be recovered the same way, and the fact that it can is the strongest check available on the whole table. AUC — the area under the receiver operating characteristic curve, the standard single-number summary of how well a score separates two labelled classes — is a rank statistic: it is the probability that a randomly drawn positive outscores a randomly drawn negative, and it never looks at the scale of the scores at all. d′ is a parametric quantity that depends entirely on the scale. Under the normal assumption the two are connected: the AUC is the probability that a normal variable falls below d′ divided by the square root of two. Substituting 1.76 gives 0.893, and substituting 3.73 gives 0.996 — both to three decimal places, matching the table. Two instruments that measure different things by different routes land on the same numbers, which says the normal assumption is holding well enough on these pairs for the derived cells to be trusted.

With the arithmetic settled, the reading is straightforward. Model A produces impressive-looking numbers. Its relevant pairs sit at 0.85, which reads as high confidence to anyone who assumes the scale runs from 0 to 1 and that 0.85 is near the top of it. Its irrelevant pairs sit at 0.78, and the two distributions overlap so heavily that no cutoff anywhere separates them cleanly — at 0.75, more than three quarters of the negatives pass. Model A is not confident. It is compressed, using a narrow band of its available range for everything it sees, which is a common property of models whose training data was dominated by a single domain or whose pooling produces vectors clustered in a small cone of the space.

Model B spreads its two classes 0.449 apart with an AUC of 0.996, which means that for essentially any relevant/irrelevant pair drawn at random, the relevant one scores higher. Its positives average 0.702, lower than Model A's negatives. A team that inherited a 0.75 cutoff and applied it to Model B would throw away two thirds of the correct passages while congratulating themselves on a strict quality bar. Both ratios can be computed from a hundred lines of evaluation code:

import numpy as np
from sklearn.metrics import roc_auc_score

def score_report(pos_scores, neg_scores):
    pooled = np.sqrt((pos_scores.var(ddof=1) + neg_scores.var(ddof=1)) / 2)
    d_prime = (pos_scores.mean() - neg_scores.mean()) / pooled
    y = np.r_[np.ones(len(pos_scores)), np.zeros(len(neg_scores))]
    s = np.r_[pos_scores, neg_scores]
    return {
        "pos_mean": round(float(pos_scores.mean()), 3),
        "neg_mean": round(float(neg_scores.mean()), 3),
        "d_prime": round(float(d_prime), 2),
        "auc": round(float(roc_auc_score(y, s)), 3),
    }

print(score_report(pos_a, neg_a))
# {'pos_mean': 0.847, 'neg_mean': 0.781, 'd_prime': 1.76, 'auc': 0.893}
print(score_report(pos_b, neg_b))
# {'pos_mean': 0.702, 'neg_mean': 0.253, 'd_prime': 3.73, 'auc': 0.996}

The pooled line is where the two spreads are combined, and it averages the two variances rather than the two standard deviations before taking a root, which is the standard pooling and is not the same as averaging the deviations. The roc_auc_score call is doing something completely different from the lines above it — it sorts every score and counts pairs — which is what makes the agreement between its output and the parametric route meaningful rather than circular.

This measurement is also the thing that makes a model comparison honest. A three-point advantage on a retrieval benchmark is a ranking result, and ranking only requires the positives to be above the negatives, not to be far above them. A filter needs distance, and the benchmark never measured it. The two models in this table could plausibly score within a few points of each other on a leaderboard while one of them supports a usable cutoff and the other does not.

Rescaling moves the number without moving the errors#

The intuitive repair, once a model swap breaks a threshold, is to rescale: if the old model's useful range was roughly 0.60 to 1.00 and the new one occupies 0.60 to 0.90, map one onto the other and the cutoff at 0.75 becomes a cutoff at 0.7125. The arithmetic is a linear interpolation — 0.75 sits three quarters of the way up the old range, and three quarters of the way up the new one is 0.7125. This feels principled and accomplishes nothing.

Any rescaling of that kind is affine — s' = αs + β with α > 0 — and the reason it changes nothing is one line. Take any two scores with s₁ > s₂. Multiplying both by a positive α preserves the inequality, since multiplying by a positive number never flips a comparison, and adding the same β to both sides preserves it too. So s₁' > s₂', for every pair, without exception. An affine map with positive slope is strictly increasing, and a strictly increasing map preserves the order of every pair of scores.

Follow that through to the filter. Whatever set of documents was admitted by threshold t on the old scale is admitted by threshold αt + β on the new one, exactly, with no document changing sides — because a document is admitted if its score exceeds the cutoff, and the map moved both quantities by the same monotone function. The positives and negatives keep their relative positions, the overlap between the two distributions is unchanged, and the false positive and false negative rates at corresponding cutoffs are identical. Rescaling changes the label on the axis.

This is the strongest form a bad defence takes, and it is worth naming as such: the error rate after rescaling is provably constant in the rescaling. No choice of α and β can move it, so no experiment on any dataset could show one choice working better than another, and the appearance of principle comes entirely from the fact that a calculation was performed.

Worse, when the rescaling is used to port a cutoff from one model to another, it imports an operating point — a specific choice of how many irrelevant results to admit in exchange for how few relevant ones to lose — that was chosen against a different overlap. The 0.75 cutoff for Model A sits where it does because of Model A's 76 percent negative admission rate — it was already a bad cutoff, and mapping it onto Model B's scale asks Model B to reproduce Model A's error profile. What you want from Model B is a cutoff chosen from Model B's own distributions: at 0.45, it admits under 6 percent of negatives and keeps 99 percent of positives, an operating point Model A cannot reach at any threshold.

Choosing the operating point is then a matter of reading it off a curve rather than guessing. Sort the labelled pairs by score, sweep the cutoff from high to low, and record precision — the share of what the filter keeps that is genuinely relevant — and recall, the share of the relevant pairs it keeps, at each step; the resulting curve says what is purchasable. For Model B on the 600 pairs above, a cutoff at 0.52 gives precision 0.90 at recall 0.94, and pushing precision to 0.95 costs a move to 0.58 and drops recall to 0.87 — a real trade, stated in units the downstream system cares about. Model A's curve has no point offering precision above 0.62 at any usable recall, which is the same fact as its AUC of 0.893 expressed in a form that makes the consequence obvious.

So the answer to "the new model's scores all fall between 0.6 and 0.9, what do I set the threshold to" is that the range does not contain the information needed to answer. A narrow range is not evidence of poor separation, because separation is measured relative to within-class spread; a model could place positives at 0.88 ± 0.005 and negatives at 0.65 ± 0.008 inside that same range and be nearly perfect. Run those hypothetical figures through d′ and the separation is enormous, because the gap is dozens of times the spread — the same span of the number line, an entirely different instrument. The threshold comes from the labelled pairs, chosen at whatever operating point the downstream consumer needs — a language model that hallucinates when fed irrelevant context, stating invented details in the same confident register as sourced ones, wants high precision; a human reviewing a shortlist wants high recall — and it is refit whenever the model, the metric, the normalisation or the chunking changes, because all four move the distributions.

Instruments that do not depend on a fixed line#

Even a correctly fitted cutoff is a blunt tool, because it applies one global line to queries of very different difficulty. A query with an exact answer in the corpus produces a top score of 0.88; a vague or out-of-domain query produces a top score of 0.61 with everything below it clustered within 0.02. A fixed line either lets the second query through wholesale or blocks the first query's weaker but genuinely useful supporting passages.

The relative gap approach keeps results while each score stays within some fraction of the top one, or cuts at the largest consecutive drop in the sorted list — if the scores are 0.88, 0.86, 0.85, 0.61, 0.60, the break after the third entry is doing more work than any absolute number could. Read that list as gaps rather than levels and the reason is plain: the first three steps are each a couple of hundredths, and the fourth is more than ten times either of them. No fixed line anywhere on the scale encodes "cut where the steps get large", because the position of that break moves with the query. It costs nothing and it fails gracefully into "keep everything" when the list is flat, which is why it is usually combined with a loose absolute floor rather than used alone.

Calibration converts the score into something that means the same thing across models. Fit a one-feature logistic regression, which fits an S-shaped curve running from score to probability, or isotonic regression, which fits any increasing step function at all, if the relationship is not monotone-sigmoid, on exactly the labelled pairs already collected:

from sklearn.linear_model import LogisticRegression
import numpy as np

X = np.r_[pos_b, neg_b].reshape(-1, 1)
y = np.r_[np.ones(len(pos_b)), np.zeros(len(neg_b))]

calibrator = LogisticRegression().fit(X, y)

def p_relevant(score: float) -> float:
    return float(calibrator.predict_proba([[score]])[0, 1])

for s in (0.30, 0.45, 0.60, 0.75):
    print(f"cosine {s:.2f} -> P(relevant) = {p_relevant(s):.2f}")

# cosine 0.30 -> P(relevant) = 0.04
# cosine 0.45 -> P(relevant) = 0.31
# cosine 0.60 -> P(relevant) = 0.84
# cosine 0.75 -> P(relevant) = 0.98

Those four outputs are worth reading against Model B's distributions rather than taken as given. The steepest part of the curve falls between 0.45 and 0.60, which is where the two classes overlap: below it almost everything is a negative, above it almost everything is a positive, and the region in between is where the probability has to move fastest. The calibrator is not adding information, it is re-expressing the separation the model already had in a unit that means the same thing whatever the model's band happens to be.

The threshold now lives in a unit a product owner can reason about — keep passages more than 70 percent likely to be relevant — and it survives a model swap, because swapping the model means refitting the calibrator on the same labelled pairs and the downstream constant stays where it is. That is the difference from rescaling, and it is worth being exact about: rescaling is a monotone map fitted to nothing, so it cannot change which pairs are confused; calibration is a monotone map fitted to the labels, so while it also cannot change which pairs are confused, it changes what a given number asserts about the pair. The errors are the same and the reporting is honest. Per-query normalisation attacks the same problem from the other direction, converting each candidate's score into its standing within that query's own returned set, typically a z-score against the top 50, counting how many standard deviations above that query's own mean the candidate sits, or a softmax that turns the 50 scores into shares of a single total, so that one fixed line stops competing with query difficulty.

What no instrument can supply#

Whichever instrument you choose, the distributions it was fitted against are a snapshot and they move. A corpus that grows from support articles into product documentation, a chunker retuned from 300-token to 800-token windows, a model version bumped by a hosted provider without a change to the model string you pass — each shifts the score distribution without touching a line of your code. The cheap defence is to log the score of every returned result and watch the distribution of the top-1 score per query as a production metric, because it drifts before recall does; a mean top-1 score that has slid from 0.71 to 0.64 over a month is a signal to re-measure against the labelled pairs, and it costs one histogram in whatever dashboard you already have. The labelled set itself deserves the same treatment as a test fixture, versioned and re-sampled occasionally, since a set collected before the corpus doubled is calibrating for a system that no longer exists.

None of these instruments manufactures separation that the embedding model did not produce. With Model A's AUC of 0.893, no cutoff, no calibration and no per-query normalisation delivers a filter that is both precise and complete, because the two distributions genuinely overlap and the information required to tell the classes apart is not in the score. That is the honest limit of everything in this chapter: each instrument is a different way of reading one number, and no reading recovers what the number never carried. Calibration will faithfully report that a 0.80 score means a 55 percent chance of relevance, which is honest and useless as a gate.

The remedy at that point is a different embedding model, or a cross-encoder reranker scoring the top 50 candidates by reading query and passage together rather than comparing two independently produced vectors — which is a more expensive computation and a much sharper one, and reranker scores need their own distribution measurement before they get a threshold of their own. That last clause is not a formality. A reranker produces a different score distribution with a different shape, often unbounded and asymmetric, so every argument in this chapter applies to it from the beginning, with none of the numbers carried over.

So a threshold is a measurement, refit whenever the thing it measures changes, and the four rows that decide whether one is possible at all are two class means and two standard deviations. Everything else in that table — the gap, the separation, the AUC, the admission rates — falls out of those four, which is why the measurement is worth an afternoon rather than a project. What it will not tell you is a number to reuse anywhere else, and the moment it is treated as one, it has become the constant it replaced.

What all of this gives you is a set of numbers that are attached to something: a metric chosen because a model card named it, vectors whose norms you have actually looked at, an index whose agreement with brute force you have measured, and scores whose meaning you have established on labelled pairs rather than inferred from the fact that they end in a decimal point. That is the state a retrieval pipeline has to be in before anything built on top of it — chunking strategy, hybrid search, reranking, evaluation harnesses — can be tuned against evidence, because until then every change you make is being scored by an instrument you have not checked.