The afternoon answer needs its comparison slice to already exist

The message arrived at 09:12 on Thursday: saw the announcement, are we switching? The model had gone live on Tuesday afternoon, two people on the team had already tried it in a scratch file, and the engineering lead had until roughly Friday evening before the absence of an answer became its own answer.

There are two versions of what happens next, and the difference between them was decided months earlier. In the first, someone spends Thursday wiring the new model into a branch of the production service, runs a few dozen real requests through it on Friday, reads the outputs, and reports that they look sharper on the harder tickets and that latency feels about the same. That report is an anecdote with a sample size of forty and no measurement of cost or tail latency, and it will be argued about for three weeks because there is nothing in it that anyone can check. In the second, a script that already exists is pointed at a new adapter, runs for eleven minutes, and emits a table that puts the candidate next to the incumbent on the same cases, the same prompt lengths and the same four axes. The second version takes an afternoon, and none of that afternoon is spent deciding what to measure.

Four axes, fixed in advance#

The comparison slice is a small standing artefact with four parts, and its defining property is that all four are specified before any particular model exists, so that no part of the specification can be chosen after you have seen which way it points.

Quality comes from the suite built in the previous chapter, run with a candidate adapter and reported as a weighted pass rate, which is the share of cases passed with each case counted according to the weight in its file, so that the incident-derived ones count triple. Nothing in the slice re-derives quality; it inherits it, which is why the suite's provider-neutral Output type matters.

Latency is measured at your prompt lengths, not at anyone else's. Published throughput figures are quoted at short prompts under light load, and the number that governs your user experience is time-to-first-token (TTFT), the delay between the request leaving and the first word appearing on the user's screen, measured at whatever your prompt actually looks like after retrieval has stuffed twelve documents into it. For the service in question that is a mean of 8,400 input tokens and 320 output tokens, so the slice replays a fixed set of 90 real prompt payloads at that profile, at a concurrency of 8, and reports p50 and p95 for both TTFT and total completion. Those two are the median and the ninety-fifth percentile: p50 describes the typical request, and p95 describes the slowest one in twenty, which is where complaints come from and which averages hide. Running that at concurrency 1 from a laptop measures a condition your production never experiences.

Cost is per task, at your token profile, not per million tokens. The conversion is arithmetic and it is the step that most often reverses a conclusion, because a model that is 40% cheaper per output token can be more expensive per task if it is more verbose or if it burns reasoning tokens you are billed for.

Known failure cases are the fourth axis and the one most often left out. These are the fourteen cases in the suite that come from incidents — the summary that dropped the refund window, the tool call that looped, the extraction that hallucinated an order ID that resolved to a real order belonging to someone else. A candidate that scores two points higher overall while failing three of the fourteen is not an improvement, because those three represent failure modes that have already cost the company something and that the current model was specifically shown not to have.

SLICE = {
    "quality":  {"cases": "suite@v9", "runs": 3, "temperature": 0},
    "latency":  {"payloads": "fixtures/latency/90.jsonl",
                 "profile": {"input_tokens": 8400, "output_tokens": 320},
                 "concurrency": 8, "report": ["ttft_p50", "ttft_p95",
                                              "total_p50", "total_p95"]},
    "cost":     {"basis": "usd_per_task", "profile_source": "prod_7d_median"},
    "failures": {"cases": "suite@v9 where provenance ~ 'INC-'", "must_pass": "all"},
}

Writing the slice down as a structure rather than as a habit is what makes it survive the person who invented it going on holiday, and it is what stops the axes quietly expanding on the day of a decision. Somebody will want to add a fifth axis in the middle of the Friday run, usually because the candidate is losing on one of the four, and the answer is that the fifth axis can be added for next time.

The Friday run, and what it actually produced#

The afternoon breaks down as 25 minutes writing the adapter, 11 minutes of wall-clock for the candidate run and another 11 for the incumbent control, about 30 minutes reading the failures rather than the aggregates, and 45 minutes on the memo. That leaves slack in a four-hour window, and the slack is what absorbs the thing that always goes wrong — a rate limit on a new account, an unfamiliar parameter name, a tokenizer difference that makes the token counts non-comparable until you fix the accounting.

The 30 minutes spent reading failures is the part that gets skipped under time pressure and pays back the most. An aggregate tells you a candidate is worse; the individual failing cases tell you whether it is worse in a way that a prompt change would fix. Two of the three regressions here were the model ignoring a length instruction, which is a prompting problem, and the third was a genuine factual drop. That distinction changes what the memo recommends and it is invisible from the summary row.

The incumbent's rows were already in the results table because the suite runs on a schedule, which removes the single largest time cost of a comparison — measuring the thing you already use. Running the incumbent afresh on the same afternoon is worth doing anyway as a control, and it takes eleven minutes, so it is cheap insurance against a provider-side change nobody announced.

The cost row is the one worth walking through, because the headline prices suggest a saving of roughly a third and the actual saving is a third of that. The incumbent costs 8,400 × $3.00 / 1,000,000 = $0.0252 for input and 320 × $15.00 / 1,000,000 = $0.0048 for output, giving $0.0300 per task. The candidate costs 8,400 × $2.50 / 1,000,000 = $0.0210 for input, and its longer answers cost 505 × $10.00 / 1,000,000 = $0.0051, giving $0.0261. The verbosity eats most of the discount, and it would have eaten all of it at a longer output profile. A per-million-token comparison would have reported a 33% saving on output and been wrong about the bill by $9,600 a month.

Verbosity is also not purely a cost question, because 505 tokens at the candidate's generation rate is what produces the p95 completion time of 9.8 seconds against 6.4. Two of the four axes moved for the same underlying reason, which is the sort of thing a fixed slice surfaces and a set of separate ad-hoc measurements does not.

Three details make the latency column trustworthy or worthless, and all three have to be fixed in the slice specification rather than decided on the day. Streaming and non-streaming calls produce completely different time-to-first-token figures, so the slice measures whichever one production uses and says so. A brand-new API key sits in a lower rate-limit tier than an established one, and hitting that tier at concurrency 8 produces retries that inflate p95 by seconds and describe your account's provisioning rather than the model — checking the retry count in the run log is a thirty-second step that has saved several teams from a wrong conclusion. Region matters too, since a candidate served from a different continent carries a round-trip penalty that has nothing to do with the model and everything to do with where it is available, which is a real cost and belongs in a different row of the memo than generation speed.

The performance-engineering habit this borrows from is worth naming, since it is the same discipline applied to a different system. Nobody credible evaluates a database by running one query on their laptop and reporting that it felt fast; they build a load profile from production traffic, fix it, and re-run it against every candidate under identical conditions, because the point of fixing the profile is that differences between runs become attributable to the system rather than to the harness. A model comparison has exactly the same structure, and the reason it is so often done by vibes is that the harness feels expensive to build until the first time you need it in 36 hours.

What the slice will not tell you#

It does not cover the operational questions, and those have blocked more migrations than quality ever has. Data residency and retention terms, whether the provider will sign your DPA — the data processing agreement your own customers' contracts oblige you to have in place with any company that handles their data — rate limits at your peak concurrency, region availability, the length of the deprecation window on the model you would be depending on, and whether your enterprise customers' contracts enumerate approved subprocessors are all binary blockers that no amount of eval running will surface. They belong on a separate checklist that gets walked in parallel, and a candidate that wins every axis in the table above is still not adoptable if it cannot be served in the region where your regulated customers' data has to stay.

It does not measure the migration cost either. The slice runs the candidate against the existing prompts, which is the correct comparison for a switching decision and an unfair one to the candidate, because a model that responds better to a different prompting style will look worse than it is. The honest way to report that is as a stated limitation with a rough estimate attached — a fortnight of prompt work might recover some of the deficit — rather than by quietly re-tuning the candidate's prompts and comparing a tuned system against an untuned one.

And it does not decide anything. The table shows quality up 2.8 points, cost down 13%, tail latency up 96%, and two incident cases regressed, which is a genuine trade rather than a verdict. Somebody in the room will argue that 2.8 points of quality is worth five seconds at p95, and somebody else will argue that the two failing incident cases are disqualifying, and both of those are defensible readings of the same numbers. Settling that argument on the Friday means settling it under the influence of a shiny release and a CEO who has already read the launch post, which is the worst possible moment to be deciding how much latency your users will tolerate.