A latency win that moves no user milestone is a cost increase

The quarter-close deck had four numbers on it. Time to first token was down 70%, from 2.4 seconds to 0.72. p95 completion time was up 15%, from 11.8 seconds to 13.6. Cost per request was up 20%, from $0.043 to $0.052. The thumbs-down rate was 2.1%, unchanged. The team's reading was that the quarter delivered a large improvement in perceived latency, a small regression in a metric users do not observe directly, a manageable cost increase, and no quality harm, and the request was to call it a success and fix completion time next quarter.

Every number in that deck is accurate. The reading of them is wrong in three separate places, and the third one is the reason the first two were not caught in week four instead of week thirteen.

What was actually bought, per surface#

Time to first token improved for 100% of traffic, and traffic on this product splits roughly 35% conversational surfaces where a human reads the answer as it streams and 65% surfaces where the output is a JSON payload consumed by the UI or handed to a downstream call. On that 65%, a first token arriving 1.7 seconds earlier is an event nobody observes, and the only milestone in play is completion, which got 1.8 seconds worse. Two thirds of the product's requests are strictly worse than they were in January.

The conversational third did get a real improvement, and it also absorbed the completion regression, because a chat user waits for the whole answer before they can act on it in every interaction that is not pure reading — following an instruction, copying a value, deciding whether to escalate. The claim that users never observe completion time directly only holds for a user who reads along and stops when they have understood, and that is one behaviour among several rather than the definition of chat.

Breaking the quarter's five changes out against the three numbers makes the arithmetic legible, and this table is the artefact the review needed and did not have:

Two rows delivered a milestone improvement at zero cost and zero regression. One row — predicted outputs enabled without a gate — bought no first-token improvement at all, cost $0.003 a request, and added half a second to p95 completion, because it was left on for the restructure-shaped requests where the prediction is rejected wholesale. That row is a pure loss, and it was invisible for a quarter because nothing in the reporting attributed the aggregate numbers back to the changes that produced them.

Why the unchanged thumbs-down rate proves nothing#

The third error is treating flat feedback as evidence of no harm. The product served about 180,000 assistant requests in the quarter. Explicit feedback was left on 0.9% of them, which is 1,620 ratings, of which 4.0% were negative — 65 thumbs-down in three months.

Ask what it would take for that instrument to detect a 15% relative worsening in the negative rate, from 4.0% to 4.6%. For a two-proportion comparison at 5% significance and 80% power — a test that would flag a difference this large four times out of five when it is genuinely there, while raising a false alarm on identical populations once in twenty — the sample needed per arm is roughly

p, delta = 0.04, 0.006          # 4.0% -> 4.6% is a 15% relative shift
n = 16 * p * (1 - p) / delta**2
print(f"{n:,.0f} ratings per arm")   # 17,067

Seventeen thousand ratings per arm against 1,620 collected in total. The instrument is short by a factor of twenty, so an unchanged thumbs-down rate is exactly what you would expect to see whether the quarter harmed quality or not. Sparse feedback cannot detect a 15% shift, and reporting it as though it could turns absence of evidence into a claim.

Sparsity is the smaller of the two problems. Ratings are self-selected, and the selection runs towards users who are either delighted or angry enough to click. The user who waits 13 seconds, closes the tab, and does the task by hand leaves no rating at all, so the population that produces the denominator is the population that stayed. A regression that increases abandonment can lower the observed thumbs-down rate while making the product worse, which is the case where flat feedback is actively misleading rather than merely uninformative. Detecting a change of this size needs the graded offline evaluation on a fixed workload, plus behavioural signals — abandonment, retry rate, time to first copy — that every session emits whether the user opts in or not.

The number that should have existed in week one#

The single artefact missing from the whole quarter is a declared target: which milestone, on which surface, to what value, with what may not get worse in exchange. Written as configuration and evaluated on the fixed workload before rollout, it looks like this:

# latency-budget.yaml — declared before the work starts, evaluated per change
objective:
  surface: chat
  milestone: time_to_first_useful_content   # client-side, p95
  baseline_ms: 4100
  target_ms:   2000

guardrails:            # any change that breaches these does not ship
  p95_completion_ms:
    baseline: 11800
    max:      12000    # ~2% headroom, not 15%
  cost_per_request_usd:
    baseline: 0.043
    max:      0.046    # 7%
  offline_quality:
    suite: eval/assistant-500
    min_pass_rate: 0.94   # baseline 0.945, one point of slack

workload: fixtures/traffic-mix-2025q1.jsonl   # 500 recorded requests, frozen

Three properties make that document do work the four-number deck could not. It names one objective, so a change that improves nothing on that line is not latency work regardless of what else it moves. It names guardrails with numbers, so "cost per request up 20%" is a breach in the week it happens rather than a line item at quarter close. And it pins the workload, so streaming, prediction, prefetch and skeleton-then-fill are all measured against the same 500 requests and their results are comparable to each other rather than to four different demos.

This is not the service-level objective from the metrics work, and confusing the two produces a document that is bad at both jobs. An SLO is a reliability commitment over a rolling window with an error budget that gets spent by incidents. This is a design constraint on a change, evaluated once, before rollout, on a frozen workload. The SLO tells you whether the service is healthy; the budget tells you whether a specific change is allowed to ship.

What would have caught it in week four#

A budget only works if something evaluates it, and the evaluation has to be attached to the change rather than to the calendar. The mechanism that fits an ordinary deployment pipeline is a gate that replays the frozen workload against the candidate build and refuses the rollout on a guardrail breach. It runs before the canary, which is the small slice of live traffic a new build is allowed to serve while the old one still handles the rest:

# runs on every candidate before the canary is promoted
latency-gate \
  --budget latency-budget.yaml \
  --workload fixtures/traffic-mix-2025q1.jsonl \
  --candidate $BUILD_SHA \
  --baseline  $LAST_PROMOTED_SHA

objective   ttfuc_p95            4100ms -> 2380ms   (target 2000ms)   PROGRESS
guardrail   completion_p95      11800ms -> 12410ms  (max 12000ms)     BREACH
guardrail   cost_per_request     $0.0430 -> $0.0475 (max $0.0460)     BREACH
guardrail   eval pass rate         94.5% -> 94.1%   (min 94.0%)       OK
result: 2 breaches — rollout blocked

Run per change, that output names the culprit while there is still one candidate to blame. Run per quarter against an aggregate, it names nothing, which is how a $0.009 cost increase and 1.8 seconds of completion regression accumulate across five separate merges each of which looked defensible in isolation.

The workload file is the piece teams skip and then regret. Five hundred recorded requests, frozen, with the real production mix of surfaces and answer lengths, means that streaming, prediction, prefetch and skeleton-then-fill produce numbers that can be subtracted from one another. Without it, each change is measured on whatever traffic happened to be flowing that week, and a comparison between two of them is a comparison between two Tuesdays.

Behavioural signals belong in the same gate, because they are the ones that would have caught the quality question the thumbs could not. Abandonment rate, retry rate, and time from render to first copy are all emitted by every session, need no opt-in, and have the sample size that explicit feedback lacks: 180,000 requests rather than 1,620. A 15% shift in abandonment on 180,000 events is detectable in a day.

The answer to "keep everything and fix completion next quarter"#

The proposal to keep all five changes and repair the regression later assumes the regression is separable from the changes that caused it, and the attribution table says it is not. Every millisecond of the 1.8-second completion regression and every cent of the $0.009 cost increase came from four of the five rows. Next quarter's completion-time work would be spent undoing this quarter's latency work, which means both quarters bought the two rows that were free.

The response is to rank the rows by milestone moved against guardrail spent and revert from the bottom. Turning off predicted outputs except where the instruction is a localised edit removes $0.003 and 0.5 seconds and costs nothing, because that row was never buying a milestone. Moving the prefetch trigger from document-open to hover-or-dwell removes about $0.003 and 0.2 seconds while keeping roughly 86% of its latency win. Restricting skeleton-then-fill to the panels with a fixed section schema removes 0.4 seconds of the 0.6 and keeps the improvement where the shape is known in advance. Those three moves put p95 completion at 12.1 seconds and cost at $0.0455, inside the guardrails above, with 1.4 seconds of the 1.68-second first-token improvement intact.

Three reverts is a smaller-sounding outcome than five ships, and the quarter is a better quarter after them than before, because what remains is the subset that moved a milestone somebody waits on. It is worth being direct with the team about why the reverts are not a judgement on the work: nothing in the attribution table was an unreasonable thing to try, and the failure was that nothing said in advance which of them would count as having worked.

What is left after that is a genuine open question rather than a mistake: the conversational surface still has a p95 time to first useful content of about 2.6 seconds against a 2.0 target, and none of the levers in this topic closes that gap without spending a guardrail. Closing it means going after the work itself — a shorter prompt, less retrieved context, a smaller model where the evaluation supports it, a faster deployment tier — which is a different kind of project with a different kind of risk.

The habit to carry into that project is the one this quarter lacked. Before any change, name the surface, the milestone, and the number you are trying to reach; name what may not get worse and by how much; freeze a workload that both will be measured on. Then instrument at the client, read the tail rather than the median, and attribute every result back to the change that produced it, so that at the end of the quarter the question "what did this buy" has an answer per row rather than an aggregate that can be read four different ways.