Quality gain and latency cost are one measurement, not two

Adding a second-stage model to a search pipeline changes two things that are usually owned by different people. It changes the ordering of results, which is measured on a labelled set of queries with human relevance judgments and reported as a metric such as nDCG@10. It also changes how long a search takes and what it costs to serve, which is measured on live traffic and reported as latency percentiles and a monthly bill. Both measurements are routine, and neither one is hard.

What is hard is putting them beside each other, because the two live in different tools on different schedules. A joint report is the artefact that fixes this: one table whose rows are candidate configurations and whose columns are relevance, latency at the percentiles the product has committed to, and spend per query, all produced from the same runs against the same serving path. The rest of this chapter is what has to be true of an experiment for that table to be worth reading.

One experiment with two inputs#

The measurement needs both a labelled set and real traffic, because neither one can produce the other's numbers. The judged set has relevance labels and a query distribution that a human curated, which makes it right for quality and wrong for latency — 200 queries run in a loop on an idle endpoint tell you nothing about what happens when 40 of them arrive in the same second. Production traffic has the arrival pattern, the concurrency, the passage-length distribution and the cache behaviour, and it has no labels at all.

So run both against the same deployed configuration, in the same session, and emit one row. The arms are configurations of the actual serving path rather than notebook code, which matters because a reranker called through a local predict() and a reranker called through an HTTP endpoint behind a load balancer are not the same system.

import time, statistics, concurrent.futures as cf

def evaluate_arm(arm, judged_set, qrels, replay_queries, concurrency=40):
    """arm: a callable hitting the real serving path with a config flag set."""
    # Quality: labelled set, sequential, correctness only.
    ndcgs = []
    for qid, q in judged_set.items():
        docs = arm(q)
        ndcgs.append(ndcg_at_k([qrels[qid].get(d["id"], 0) for d in docs[:10]],
                               list(qrels[qid].values()), k=10))

    # Latency + cost: replayed production traffic at production concurrency.
    lat, pairs = [], 0
    def one(q):
        t0 = time.perf_counter()
        docs = arm(q)
        return time.perf_counter() - t0, arm.last_candidate_count
    with cf.ThreadPoolExecutor(max_workers=concurrency) as pool:
        for dt, n_pairs in pool.map(one, replay_queries):
            lat.append(dt * 1000)
            pairs += n_pairs

    lat.sort()
    pct = lambda p: lat[int(len(lat) * p) - 1]
    return {
        "ndcg@10": statistics.mean(ndcgs),
        "p50_ms": pct(0.50), "p95_ms": pct(0.95), "p99_ms": pct(0.99),
        "rerank_pairs_per_query": pairs / len(replay_queries),
        "usd_per_1k_queries": cost_model(pairs) / len(replay_queries) * 1000,
    }

Two details in that function carry the whole design. The quality loop runs sequentially, because correctness does not depend on how many other requests are in flight and running it under load would only add noise. The latency loop runs through a thread pool sized to production concurrency, because a percentile is a statement about a queue and a queue with one customer in it has no tail. Reading the two loops as one function is the point of writing them as one function.

The replay set for this system was 84,000 queries sampled from one week of logs, weighted to preserve the hourly arrival curve so that the busy-hour concurrency showed up rather than being averaged away. Sampling uniformly across a week gives a smooth load that no hour of the week actually experienced, and the tail is a property of the busy hour.

The measurement that produced the original notebook's latency figure is worth naming, because it is the default one and it is wrong in a specific, repeatable way. Scoring 200 queries × 50 candidates as a single predict() call with batch_size=64 on a warm local GPU reports about 6 ms per query of amortised model time, and that number is honest about the model and silent about the system. It has no network hop, no JSON serialisation, no other tenant on the GPU, no arrival pattern, and a batch composition chosen by the benchmark rather than by whatever requests happened to be in flight.

The batch composition is the part that does the most damage, and it is visible in the call's own arguments. Fifty candidates per query is fewer than the batch size of 64, so a request that scores one query's candidates cannot fill a single batch; it submits a partial one and pays the fixed per-batch overhead on it. A bulk call over the whole judged set has ten thousand pairs to draw from and fills every batch completely, so the same model on the same card does more work per unit of time than any individual request will ever see. Any latency figure gathered without concurrency is a throughput figure wearing a latency label, and this one is a throughput figure that also had the batching arranged in its favour.

One more property of the harness decides whether the rows can be compared to each other: the arms have to run in the same window rather than one after the other. The first version of this experiment measured arm A over lunchtime and arm B two hours later, and arm B's p95 came out 71 ms worse than the interleaved re-run eventually showed it to be, because the afternoon peak had started and the shared search cluster was serving more traffic from other services. Interleaving the arms request by request, or at worst alternating in five-minute blocks, puts both arms under the same neighbours, the same cache warmth and the same autoscaling state. It costs nothing to implement — a modulo on the request index — and it removes the most common way a latency comparison manufactures a difference that belongs to the clock.

The row that settles the argument#

Arm B is the configuration that shipped and was rolled back, and the table explains the rollback in one glance: 526 ms at p95 against a 500 ms objective. Arm C buys one more point of nDCG@10 for another 177 ms and double the spend, which is not a decision anyone needs to deliberate over. Both of those figures come out of subtraction along a column: 703 minus 526 is the 177 ms, and 0.72 against 0.71 is the single point, with the spend column doubling because the pairs column did.

Arm D shows the calibrated threshold from the previous chapter costing nothing in latency, because all fifty candidates were scored either way and the bar is applied afterwards. The table says so in a way worth reading deliberately: D's pairs-per-query and dollars match B's exactly, and its three latency percentiles sit one to three milliseconds above B's, which is measurement noise rather than work. The quality effect of arm D lives in ungrounded answers, which is not a column in this table and needs one of its own.

Arm E is the row that resolves the standoff. Running three warm replicas instead of one, so requests stop queueing behind each other at the busy hour, brings p95 to 288 ms and p99 to 402 ms while leaving p50 essentially unchanged at 138 ms. It costs $1.10 per thousand queries instead of $0.42, because two of the three replicas are idle much of the day.

The shape of arm E against arm B is the diagnosis, and it is the one reading that tells you where the milliseconds were. A median that does not move while the ninety-fifth and ninety-ninth percentiles collapse is the signature of queueing rather than of computation: the typical request was never waiting for anything, and the slow ones were waiting for a replica that was busy. Adding replicas can only help a request that was queued, so the fact that it helps this much is evidence about what the tail was made of. The argument that took a week to have was framed as quality against latency, and the actual choice available was quality against money, which is a much easier conversation and one that a $700-a-month line item resolves.

Writing an exchange rate between the columns is tempting and usually not how the decision gets made. Arm B buys 0.20 of nDCG@10 for 340 ms of p95, which is 1.7 ms per thousandth of a point, and that ratio is uninterpretable because nobody knows what a millisecond of p95 is worth in this product. The version of the question that has an answer treats latency as a constraint rather than as a currency: given a hard p95 bound of 500 ms and a cost ceiling, which arm has the highest nDCG@10? That reading turns a five-column table into a filter followed by a maximisation, and it is why arms B and C are eliminated by a number in a service objective document rather than by anyone's judgement.

The columns therefore have roles, and stating them before the experiment stops the discussion drifting. Relevance is the objective. Latency percentiles are constraints with values inherited from the product's SLO, the service-level objective the team has committed to. Spend is a constraint with a value inherited from the budget. Cost per query is the only one that is genuinely negotiable in the room, since the others were agreed elsewhere. A report that presents all five as peers invites everyone to weight them by preference.

One mean hides who paid and who was paid#

A single nDCG figure over the judged set spreads the gain evenly across queries that did not share it evenly, while the latency and spend columns are charged to every query regardless. Breaking arm B down by query class, using the same 200 judged queries tagged by the categories support already uses for tickets, changes what the table is asking.

The share column adds to a hundred percent exactly, with nothing left over, and that is the first thing to check on a breakdown like this. It means the four rows are a partition of the traffic rather than a selection of interesting cases, so there is no unlisted fifth class quietly holding the queries that did badly, and any statement of the form "most of the traffic" can be settled by adding rows.

Forty-one percent of the traffic gained 0.01 and paid a p95 of 498 ms for it, because a navigational query's answer was already at rank 1 and reordering the list beneath it changes nothing that reaches the user. That 498 ms is the lowest class p95 in the table and still sits within a rounding distance of the 526 ms the configuration costs overall, so the largest class in the system is paying essentially the full price for essentially none of the benefit.

The two classes that gained most, at 0.35 and 0.27, are 42% of traffic between them — the second and third share figures added together, and the arithmetic is worth doing because it changes the recommendation. Two rows of the table are almost as much traffic as the row that gained nothing, so this is not a case of a tiny segment benefiting. That distribution is an argument for routing rather than for a global flag: a cheap classifier, or a rule as crude as "skip the reranker when the top embedding score exceeds a threshold and the query is under four tokens", removes the reranker from 41% of requests.

The saving from that follows directly. If forty-one percent of requests stop being reranked, fifty-nine percent still are, and fifty-nine hundredths of $0.42 is just under $0.25 per thousand — which is the figure to expect, alongside the load that comes off the endpoint that was producing the tail in the first place. The spend column scales with the share of traffic that reaches the model because every rerank costs the same, and that proportionality is what makes routing a lever with a predictable price rather than a hopeful one.

The segmented view also protects against the opposite error, which is killing a reranker whose mean gain looks unimpressive. A pipeline where the mean moves 0.04 and one 8%-of-traffic segment moves 0.30 is not a pipeline that gained nothing; it is one where the relevant question is whether that segment matters commercially. Reporting a single mean makes both mistakes reachable from the same table, and the fix is a column for share of traffic beside every quality number.

What the money column is made of#

Per-query inference spend has to be built from a utilisation assumption, and the assumption is where most cost estimates go wrong. A single A10G instance at $1.01 per hour scores about 1,500 pairs per second at 256 tokens. At 50 pairs per query that is 30 queries per second at full saturation, but an endpoint run at full saturation has a queueing tail that makes the p95 column unusable, so the planning figure is around 60% utilisation, or 18 queries per second.

Each step there is a division or a multiplication with a reason attached. Dividing pairs per second by pairs per query gives queries per second, because those are the units that cancel. Taking sixty percent of that is a choice rather than a measurement — it is the headroom the team decided to keep, and a team willing to live with a worse tail would pick a higher number and get a cheaper one. That gives $1.01 / (18 × 3600) = $0.0000156 per query, dividing the hourly price by the queries an hour contains, which is the $0.0156 per thousand you would get if the instance were busy all day at that rate.

Arm B's $0.42 reflects what actually happens: traffic is concentrated in about six hours, the instance is provisioned continuously, and the bill divided by the week's real query count is 27 times that planning figure — and further still above what a genuinely saturated instance would have cost per query. Almost the whole gap is hours in which the replica was running and nobody was searching.

The hosted API at $2.00 per thousand searches is roughly five times the single-replica self-hosted figure of $0.42 at this volume, and under twice the three-replica figure. Below some volume it is cheaper than any of them, because the self-hosted rows are dominated by provisioned idle time and the hosted row is not.

Where that crossover sits is worth deriving rather than quoting, since it is one division and every input to it is local. A provisioned replica costs its monthly bill whether it serves a thousand queries or a million, and an instance at $1.01 an hour billed continuously comes to something over seven hundred dollars a month. The hosted API charges for searches and nothing for the searches nobody makes, so the two bills meet where the metered one climbs to the fixed one, which at these two prices is a few hundred thousand searches a month rather than the 40,000 a rougher estimate suggests. Volume is the variable that flips the answer, and a build-versus-buy decision made at 5,000 queries a month should be revisited at 500,000 rather than treated as settled.

Offline agreement and the online experiment#

Everything above is offline, and offline quality is a proxy. The judged set says the ordering improved by twenty points of nDCG@10; it does not say that answer quality improved, that ticket deflection improved, or that anyone clicked anything different. A team that ships on the strength of an offline joint report has removed one failure mode and kept another, and the online experiment — same flag, real users, business metric, latency as a guardrail rather than as a headline — is what closes it.

The relationship between the two is worth stating plainly, because it is not obvious in which direction the surprises run. Offline gains of this size usually shrink online, since the judged set over-represents queries someone thought were interesting; small offline gains sometimes grow online, when the affected queries turn out to be the high-intent ones. What the offline joint report is for is deciding which arms are worth an online experiment at all, and arm C never needed one.

The joint report is also silent about one thing it looks like it should cover. Every latency number in it is measured end to end at the client, so the p95 of 526 ms is real, and none of it is attributed to any component. That distinction is easy to lose, because a table with a latency column looks like a table that explains latency, and a team that reads it as one will tune the wrong knob.

The gap that leaves is large enough to see from the numbers already on the page. The vendor benchmark for this model says 30 ms for fifty documents, arm B's p50 rose by 62 ms, and its p95 rose by 340 ms. Even the median increase is twice what the model claims to cost, and the tail increase is more than ten times it. A model that takes 30 ms cannot produce a 340 ms tail on its own, and finding out what did means taking the request apart.

Quality and cost belong in one artefact because the decision is a single decision, and every arrangement that splits them across two artefacts produces the same failure: two accurate reports, no defensible answer, and whichever number reached a dashboard first deciding what ships. One table with relevance, percentiles and spend on each row turns a week of argument into a filter and a maximisation, and it is cheap enough to produce that there is no good reason to have the argument instead.