A one-point gain that costs 2.4x the calls is a losing trade

The numbers on the screen are 92 and 93. The loop that has been in production for two months passes 92% of the 240-case suite; the orchestrator–worker version, built over three weeks and evaluated with model and prompts held constant, passes 93%. It uses 2.4× the model calls and moves p95 latency from 4.0 seconds to 7.6. Half the room reads that as a win that should ship, on the grounds that a point of quality is a point of quality and users are the ones who experience the failures. The other half reads it as an obvious no.

Both halves are arguing about a single number without having converted it into anything comparable to what it costs. That conversion is the work of this chapter, and it goes in both directions: the point has to be turned into the cases it is made of, and the multiple has to be turned into money and seconds.

What one point actually is#

On a 240-case suite, one percentage point is 2.4 cases, which follows from the size of the suite alone — a hundredth of two hundred and forty. That is small enough that the first thing to do is stop treating it as a score and go read the cases, which the paired evaluation from chapter 4 makes possible because both designs ran the same inputs.

Two of those four cells carry no information about which design is better. The 214 cases both designs pass and the seven both designs fail would appear in exactly the same place whichever arm you shipped, so the comparison rests entirely on the off-diagonal. Nineteen cases disagree. The orchestrator fixes twelve and breaks seven, for a net of five cases, which is the 1 point. Whether a 12–7 split among nineteen discordant cases is signal or noise is a question with a standard answer: under the null hypothesis that the two designs are equally good, seeing a split at least this lopsided has a probability around 0.36, which is not a result anybody should build on.

The reasoning behind that probability is worth having in plain terms, because the number decides how much of your week the gap deserves. If the two designs are genuinely equally good, then for each of the nineteen cases where they disagree, which one gets it right is a coin flip. So the question becomes how often nineteen coin flips land at least twelve-to-seven in one direction or the other, and the answer is about a third of the time. That is common enough to happen by accident on a quiet afternoon. Running three repeats per case would tighten it; on this project two of the seven regressions flipped between repeats, which tells you the underlying difference is smaller than the headline.

The residual on that test is the part people over-read. A probability around a third does not establish that the designs are equal; it establishes that this suite, at this size, cannot tell them apart. Nineteen discordant cases is a small instrument, and a real difference of half a point would be invisible to it. What follows is that the decision has to be made on the other columns of the ledger, and not that the two arms have been proven identical.

The twelve fixed cases are worth reading rather than counting. Nine were multi-part questions where the loop answered the first part well and ran out of attention for the third, which is a real weakness of a single context, and it is also the kind of weakness that a longer iteration cap or a better output schema might address without any coordination. The seven regressions were all the same failure: a worker returned nothing, and the aggregation step wrote around the hole in the way chapter 5 described.

The other side of the ledger, in currency#

The call-count multiple has to be converted into money and seconds before it can be traded against 2.4 cases.

PRICES = {"in": 3.00 / 1_000_000, "out": 15.00 / 1_000_000}   # $ per token
REQUESTS_PER_MONTH = 120_000

def monthly_cost(calls, in_tokens_per_call, out_tokens_per_call):
    per_request = (calls * in_tokens_per_call * PRICES["in"]
                   + calls * out_tokens_per_call * PRICES["out"])
    return per_request, per_request * REQUESTS_PER_MONTH

loop  = monthly_cost(calls=5,  in_tokens_per_call=4_500, out_tokens_per_call=300)
orch  = monthly_cost(calls=12, in_tokens_per_call=3_000, out_tokens_per_call=350)
# loop -> $0.0900/request, $10,800/month
# orch -> $0.1260/request, $15,120/month   (+$4,320/month, +$51,840/year)

The shape of that function is the whole pricing model, and it is worth walking once on the loop's numbers so the multiples later in the chapter are readable rather than quoted. A request's bill is its calls multiplied by what one call costs, and one call costs its input tokens at the input price plus its output tokens at the output price. Five calls at four and a half thousand input tokens each is twenty-two and a half thousand input tokens, and at three dollars a million that is a little under seven cents. Five calls at three hundred output tokens is fifteen hundred output tokens, and at fifteen dollars a million that is a little over two cents. Together they are the nine cents a request that the comment records, and multiplying by a hundred and twenty thousand requests gives the monthly figure.

Two properties of that formula matter more than the arithmetic. Output tokens cost five times what input tokens cost here, so a design that talks more is penalised harder than one that reads more. And the call count multiplies both terms, which is why call count is the quantity people quote — but it multiplies a per-call context that is itself a design choice, and that is where the two arms differ.

The orchestrator's per-call context is smaller, because each worker gets a narrow slice rather than the loop's accumulating history, so 2.4× the calls comes out at 1.4× the cost rather than 2.4×. That is a detail worth computing rather than assuming, and it cuts in the orchestrator's favour. It still lands at about fifty-two thousand dollars a year for five cases out of 240 that a coin-flip test cannot distinguish from noise.

Latency is the row that decides it here. The product surface is a chat panel with a 10-second client timeout, and the distributions are not just shifted, they are shaped differently: the loop's p99 is 6.5 seconds, the orchestrator's is 14.2, because its tail is the maximum of five concurrent workers plus an aggregation call rather than the sum of a few sequential ones. About 2.1% of orchestrator requests would exceed the client timeout and show the user an error, against 0.1% today. Twenty-one failed requests in a thousand, traded for five extra passes in 240 cases, is a trade that loses on its own terms before anybody mentions operations.

The two latency increases are not the same size, and the difference is a fact about the shape rather than a rounding artefact: p95 rises by about ninety per cent while p99 rises by about a hundred and twenty. The reason is in the word maximum. A request's time is the slowest of its five workers, and the slowest of five draws lands further out than any single draw would, with the effect growing as you move into the tail — one worker having a bad second is enough to make the whole request slow, and with five workers there are five chances of that per request. A design whose latency is a sum of sequential steps has no such amplification, so its own tail sits closer to its median.

The timeout row is the one that catches teams who are watching the wrong quantity. It is natural to track mean or median latency, and by that measure the orchestrator is a few seconds slower and still comfortably inside a ten-second budget. What a median holds constant is the experience of the typical request. What varies underneath it is the share of requests that fail outright, and that share is decided in the tail, where a design whose latency is a maximum over branches behaves quite differently from one whose latency is a sum. A dashboard of medians can stay flat and reassuring while one request in fifty starts returning an error.

Ship the loop. One point does not pay for 2.4× the calls, a doubled tail that puts 2% of requests past the client timeout, and four operational obligations that nobody has written code for yet.

Saying it that way — out loud, with the table on the screen — is the part that makes the decision durable, because in six weeks somebody will propose the orchestrator again and the answer will be a link rather than a re-run of the meeting. It also names the conditions under which the answer changes. If the gap were six points, or if the point were concentrated in the class of case the business cares most about, or if the twelve fixed cases were all in the adversarial slice where the failure mode is a confident wrong answer about someone's leave entitlement, the arithmetic would be different, because a point of "stops being confidently wrong" is not the same currency as a point of average quality. Set those conditions down next to the decision so the next proposal has to meet them rather than restart the argument.

What a point of offline score is worth in production#

The trade above prices cost precisely and quality loosely, which is the usual asymmetry: the inference bill arrives monthly with two decimal places and the quality figure is a percentage on a suite somebody assembled in an afternoon. Tightening the quality side means finding out what an evaluation point corresponds to in a metric the business already tracks.

For the policy assistant, the production metric was the escalation rate — the fraction of questions where the user clicked through to open a ticket with the HR team, which costs roughly 12 minutes of a specialist's time. Over four months and five releases, the relationship between the offline suite and that rate was direct enough to use: each evaluation point was worth about 0.6 percentage points of escalation, on a base rate of 9%.

That chain of reasoning converts a score into hours, and each link is a multiplication worth seeing. Six tenths of a percentage point of escalation, applied to a hundred and twenty thousand requests a month, is seven hundred and twenty escalations that do not happen. Each one costs about twelve minutes of a specialist, so seven hundred and twenty of them is a hundred and forty-four hours. At 120,000 requests a month that makes one point about 720 avoided escalations, or 144 specialist-hours, which is worth considerably more than the $4,320 of extra inference and would flip the decision — except that the 1 point here is not established, since 12 fixed against 7 broken is a coin flip, and 0.6 points of escalation on a difference that may be zero is zero.

Building that mapping takes a few releases of patience and it is the highest-leverage measurement in this chapter, because it converts every future architecture argument into the same currency. It also has a known failure: the relationship holds only while the suite's mix resembles production traffic, so it has to be re-derived when the traffic mix shifts, and a suite that has been tuned against for six months tends to drift out of correspondence quietly.

Where no such mapping exists, the honest position is that the quality side of the ledger is an estimate with wide error bars and the cost side is not, which is itself an argument for the cheaper design when the measured gap is small. That is not a general preference for simplicity; it is what asymmetric measurement precision implies.

Where this arithmetic gives the opposite answer#

The trade is domain-specific, and the same table with different numbers flips cleanly. A medical-coding pipeline that assigns billing codes to clinical notes processes 40,000 notes a month, and an accuracy point there is worth roughly $2 in recovered billing per note that changes, which puts one point at about $9,600 a month against an inference delta measured in hundreds. Latency is irrelevant because the pipeline runs as an overnight batch, and there is no client timeout to blow. On that job the orchestrator ships at one point, and it would ship at half a point, because a point of quality is money and the calls are not.

Three things moved between the two examples, and only one of them is the score. The value of a correct answer went from a fraction of a support interaction to a billing line. The latency constraint disappeared, because a batch job has no client waiting on it, which removes the row that decided the first case outright. And the volume changed. None of those is a property of the topology, which is the point: the same architecture is a losing trade in one setting and an obvious ship in another, so an architecture preference carried between projects is a preference carried without its reasons.

The general form is that quality has to be priced in the same units as the cost before the comparison means anything, and the price varies by three or four orders of magnitude across domains. Teams that skip the pricing step tend to default to whichever side of the argument they were already on.

Both of these decisions rest on a snapshot: one evaluation set, one traffic mix, one month. The set was assembled from the traffic you had, and the shape hypothesis from chapter 1 was formed from a sample of inputs that may not describe what production is actually sending you now. The instrument that catches that is not the evaluation suite, which only ever tells you about the cases you thought to write down.