The bi-encoder and the cross-encoder are two points on one curve

The planning meeting had 20,000 pairs, one engineer for six weeks, and two proposals. The first was to fine-tune the bi-encoder. The second was to train a cross-encoder reranker, argued on the grounds that it scores each pair jointly, is more accurate per pair, and requires no re-embedding, so it strictly dominates. Both models were eventually built and both helped, which makes the framing of the meeting the actual error: the two are not competing answers to one question but two stages of one pipeline, and the decision on the table was which to build in the first six weeks.

The measurements from this corpus put numbers under that. The fine-tuned bi-encoder raised recall@50 from 0.79 to 0.87 and nDCG@10 from 0.58 to 0.68. The fine-tuned cross-encoder, sitting on top of the unmodified base retriever, left recall@50 at exactly 0.79 — reranking the top 50 cannot change which documents are in the top 50 — while moving nDCG@10 from 0.58 to 0.79 and recall@5 from 0.55 to 0.71. Those are different shapes of improvement measured on the same queries, and either one alone would have been reported as a success.

That "exactly" is not an approximation and it is the hinge of the chapter, so it is worth deriving rather than accepting. Recall at depth fifty asks what fraction of the gold passages appear anywhere in the top fifty. A reranker is handed the top fifty, computes a score for each, and returns those same fifty in a different order. Reordering a set does not change which items are in it, so the count of gold passages inside the set is identical before and after, for every query, for every reranker, however good or bad it is. The quantity is therefore pinned to whatever the first stage produced and is completely insensitive to the second stage's quality. A number that does not move when the thing you are evaluating moves is not weak evidence about it — it is a measurement of something else.

The costs are not the same kind of cost#

The row that decides most projects is "time to first production traffic". A reranker is additive: it reads the candidate list, returns it reordered, and touches no stored data, so it can be deployed behind a feature flag on 5% of traffic on a Wednesday and removed on Thursday if the p99 latency is unacceptable. A new encoder is a migration, because every one of the 1.8 million stored vectors was produced by the old model and vectors from two different encoders are not comparable — cosine similarity between them is a number without meaning. Until the last chunk is re-encoded, the index is inconsistent, which is a different class of engineering problem with a different class of failure.

That asymmetry, and not the per-pair accuracy argument, is why the reranker is usually the correct first move. It is also why the accuracy argument as stated in the meeting is misleading. "More accurate per pair" is true and says nothing about the pairs the reranker never sees, and the 0.79 recall@50 in the right-hand column is the reranker's ceiling written down explicitly. If chapter 1's triage found that the gold passage is frequently absent from the top 50, the cross-encoder cannot address it at any budget.

The loop at the bottom is the shape most teams actually follow. Shipping the reranker moves precision immediately and, by raising the quality of what users click, improves the click logs that chapter 2 mines — the training set for the bi-encoder gets better because the reranker shipped. It also makes the recall ceiling visible: once ordering is good, the remaining failures are almost entirely candidate-stage misses, and the argument for the encoder work stops being a projection and becomes a measurement.

The +35 ms in the table is one point on a curve that you choose, and choosing it is the main serving decision a reranker brings. Scoring k pairs costs roughly k × 0.7 ms on an A10G at 256 tokens before batching, so k=20 is 14 ms, k=50 is 35 ms, k=100 is 70 ms and k=200 is 140 ms; batching the pairs for one query into a single forward call recovers a good part of that through GPU parallelism, bringing k=50 to about 12 ms in practice and k=200 to about 40 ms. Quality follows a flatter curve than cost: on this corpus nDCG@10 was 0.75 at k=20, 0.79 at k=50, 0.80 at k=100 and 0.80 at k=200, so everything past 50 was paying linearly for nothing. Measure that curve on your own data before picking k, because its shape is a property of how good the first stage is — a weak retriever needs more depth to have the answer available at all, which is another way the two stages trade against each other.

Setting those two columns beside each other is what makes the choice of k obvious, and the reason is that they are curves of different kinds. Cost is strictly proportional: every entry in that latency series is the per-pair figure times the depth, with no fixed overhead and no saturation, so doubling the depth doubles the bill for as long as you keep doubling. Quality is a series of shrinking gaps — four points from k=20 to k=50, one point from k=50 to k=100, nothing at all from k=100 to k=200. A quantity growing without limit against one approaching a ceiling can only cross once, and past the crossing every further step is pure cost. That is a general argument about the shape of the two curves rather than a fact about this corpus; what your corpus decides is where the crossing falls.

Capacity is the part that surprises teams whose stack had no GPU in the request path. One A10G at roughly 2,100 pairs per second sustained serves about 42 queries per second at k=50, so a service doing 300 qps at peak needs seven of them, or $3,800 a month, or a smaller and quantised model. That last option is real: a six-layer MiniLM cross-encoder exported to ONNX, a portable format that runs outside the training framework, with its weights quantised to eight-bit integers, runs about 350 pairs per second on eight CPU cores, which is 7 qps at k=50 and enough for internal tools, at a cost of one to two nDCG points from the quantisation. Whatever the deployment, the stage needs a timeout and a defined behaviour when it expires, and the correct behaviour is to return the first stage's ordering rather than an error, so that a reranker outage degrades relevance instead of taking search down.

Both capacity figures divide out exactly — twenty-one hundred pairs a second over fifty pairs a query is the 42, and three hundred and fifty over fifty is the 7 — and a reader checking them against the latency figures above will find an apparent contradiction worth resolving rather than ignoring. Forty-two queries a second implies a little under twenty-four milliseconds of GPU occupancy per query, while the batched latency quoted for k=50 is about 12 ms. Both are correct, because they measure different things. The 12 ms is how long one query waits when the card is otherwise idle and its fifty pairs can be issued as a single wide forward call. The 2,100 pairs a second is what the card sustains under continuous load, where tokenisation, queueing and imperfect batching across concurrent queries all take a share. Sizing a fleet from the idle-latency figure under-provisions it by about half; quoting the sustained figure as a latency promise overstates the delay a user sees. Capacity and single-request latency are different measurements of one device, and neither is the other divided by anything.

That fallback is also what makes the "is it even a choice" question have a real answer in the small number of cases where it is one. A team with no GPU budget, a hard 25 ms p99, or a serving environment where adding a stateful inference service is organisationally impossible is genuinely choosing, and for them the bi-encoder fine-tune is the only lever, because its cost is paid once during re-embedding and its serving profile is identical to what they run today. That is a constraint argument rather than a quality argument, and it should be stated as one.

Stacking them is not additive#

Running both fine-tuned models together gave nDCG@10 of 0.81, against 0.79 for the cross-encoder alone and 0.68 for the bi-encoder alone, on a baseline of 0.58.

Four differences can be read out of those numbers, and which baseline each one is measured against decides what it means. Against the 0.58 baseline, the bi-encoder alone is worth 0.10 and the cross-encoder alone is worth 0.21. Against the fine-tuned bi-encoder's 0.68, the cross-encoder is worth 0.11 — ten points less than it was worth on its own. Against the cross-encoder's 0.79, the bi-encoder is worth two points, against the ten it was worth on its own. The two separate gains over the baseline add to thirty-one hundredths and the combined gain is 0.23, so the pair delivers about three quarters of what two independent improvements would have delivered. Whichever model goes second contributes far less on top of the first than it did alone, and the shortfall is the same shortfall read from either end.

The overlap has a cause worth understanding. Both models were trained on the same 20,000 pairs and therefore learned the same domain distinctions from the same evidence; where the bi-encoder has already learned that a query about cabinet warmth means thermal derate rather than warranty coverage, the reranker's version of that knowledge has nothing left to correct. What the combination does deliver is the part neither can do alone: the bi-encoder raises what reaches the reranker, and the reranker orders it, so recall@5 with both reached 0.78 against 0.71 for the reranker on the base retriever. Improvements in the first stage show up in the final metric through the second stage, which means measuring the encoder change with the reranker disabled and then again with it enabled is the only way to see how much of its gain survives.

This is also where the practical order of retrieval depth gets set. With a stronger first stage you can retrieve fewer candidates for the same recall: at recall@k of 0.87, the fine-tuned encoder reaches at k=50 what the base encoder needed k=200 to reach, and reranking 50 pairs instead of 200 cuts the reranker's GPU cost and its latency by a factor of four. A bi-encoder improvement can pay for itself in serving cost through the stage that follows it, which is not a benefit anyone puts in the business case and is often larger than the quality gain.

Training both from one dataset needs one discipline to keep the comparison honest, which is that the split has to be decided once and shared. Partition the 20,000 pairs by document group into train and held-out before either model is built, train both on the same training portion, and evaluate both against the frozen judged set from stage 8.3 rather than against the held-out pairs. Letting each model be tuned against its own split produces two numbers measured on different populations, and the stacking arithmetic above becomes meaningless. The mined negatives are shared too: the same file that supplies the bi-encoder's negative column supplies the cross-encoder's zero-labelled rows, so the filtering work from chapter 6 is done once and paid for by both.

The middle of the curve, and where it is heading#

The curve has points between the two ends. Late-interaction models store per-token vectors and compute their interaction at query time, buying part of the cross-encoder's joint reasoning while keeping the document side precomputable, at a large storage multiple — the architecture was covered in stage 8.14 and its relevance here is that it is trained from the same query-passage pairs you have already assembled, with a loss shaped like chapter 4's over token-level scores.

The more useful technique for a team with both models already trained is distillation, where a cheaper model is trained to reproduce the scores of a more accurate one instead of learning from the labels directly. Score a large set of query-passage pairs with the cross-encoder — including the mined negatives, which is what makes it cheap, since those pairs are already in a file — and train the bi-encoder to reproduce the margin between the positive's score and each negative's score rather than to satisfy a binary label. Published as MarginMSE, it consistently produces stronger bi-encoders than the same pairs used with a plain contrastive objective, because the cross-encoder supplies graded supervision where the labels supplied only presence or absence.

import torch.nn.functional as F

def margin_mse(q, p_pos, p_neg, ce_pos, ce_neg):
    """q, p_*: (B, d) normalised bi-encoder embeddings.
    ce_*: (B,) cross-encoder logits for the same pairs, computed offline."""
    student_margin = (q * p_pos).sum(-1) - (q * p_neg).sum(-1)
    teacher_margin = ce_pos - ce_neg
    return F.mse_loss(student_margin, teacher_margin)

The reason that loss subtracts before it compares is the same difference-only property the contrastive objective had, arriving from the other direction. A cross-encoder's logits and a bi-encoder's cosine similarities are on unrelated scales, so asking the student to match the teacher's score would be asking it to adopt a unit it has no way to reach. Subtracting the negative's score from the positive's cancels whatever additive offset each model carries, and what survives is the quantity both architectures can express: how much better the positive is than the negative. That is why the supervision transfers between two models that share no output convention.

The distilled result is still a bi-encoder with a bi-encoder's ceiling, so it does not remove the reranker from the pipeline. It raises the floor the reranker starts from, which in the arithmetic above is where the compounding lives.

Two gains that are not substitutes#

Treating the two as alternatives assumes their gains are substitutes, and the recall column shows they are not: no reranker moves recall@50 by any amount, and no bi-encoder gets you a joint reading of query and passage over the top few. A team that ships only the reranker has capped itself at the base encoder's recall; a team that ships only the fine-tuned encoder has left the largest single nDCG gain on the table for the price of a GPU.

The belief that makes this expensive is that a very accurate reranker is a defence against a weak retriever, and it is tempting because a reranker is visibly better at judging relevance than the encoder in front of it. The quantity that refuses to move is what refutes it. Recall@50 sits at 0.79 whether the reranker is untrained, fine-tuned, distilled or absent, because a permutation of fifty documents contains the same fifty documents. So the property that decides whether a query can be answered at all is constant across the entire range of the reranker's quality, and a team improving the reranker is watching a metric that is structurally incapable of reporting the failure they have. The instrument that does report it is the triage of chapter 1, run on the candidate stage.

The cross-encoder is also not free in the way "no re-embedding" suggests. It introduces a GPU dependency into a request path that previously had none, along with a queue, a timeout policy, and a decision about what to serve when the reranker is unavailable — which should be the unreranked list, degraded rather than failed, and that fallback needs to exist before launch rather than during the first incident. Its cost also scales with traffic in a way the encoder's does not: doubling queries doubles the reranking bill, while the fine-tuned encoder's cost was paid once during re-embedding.

Which brings the sequence back to the thing that has been deferred twice. The bi-encoder's gain is real, measured, and locked behind a migration whose price has not yet been named. Naming it is what turns a good evaluation result into a shipping decision, and the result of that decision is legitimately allowed to be no.