Retries measure user effort, and latency raises effort

On Tuesday morning the regenerate rate on the research assistant went from 4.1% to 12.3% of responses and stayed there for the rest of the week. Nothing had been released: no prompt change, no model version bump, no reindex, no flag flip in the orchestration service. Thumbs-down volume was flat at about 1,200 a week on 300,000 responses. By Wednesday afternoon somebody had proposed pulling the retried requests into the evaluation suite as failures, on the reasonable-sounding grounds that eight extra points of retries is 24,000 users a week telling you the answer was not good enough.

Retrying is a user paying a cost to get a second attempt, and people pay that cost when the expected benefit exceeds it. Both sides of that inequality move for reasons that have nothing to do with the answer. If the button used to be behind an overflow menu and is now inline under every response, the cost fell. If a week of harder questions arrived because a customer onboarded a new department, the expected benefit rose. If responses now take eleven seconds instead of seven, the user has already spent long enough staring at a spinner that they are primed to reject what comes out of it, and — more mundanely — they have had time to decide the request was badly phrased and to want another go. None of these is a quality change, and all of them move the retry rate.

The order to check things in#

The ordering below is not arbitrary. It runs cheapest and most-likely-to-be-artefactual first, so that four hours are not spent modelling a metric that was double-counting events all along.

Each branch rules something out rather than merely suggesting something. If the server-side count of regenerate requests is 4.2% while the client event stream says 12.3%, the metric is broken and no theory about user behaviour applies; a mobile client shipped the previous Friday that fires the event on both press and release will do exactly this. If the spike is entirely inside one tenant, every global explanation is dead, and the question becomes what that tenant did. A frontend deploy that moved the control is a change in the cost of retrying, and it is worth noting that the resulting rate is permanently different — the old 4% and the new 12% are not comparable numbers, so the baseline has to be re-cut rather than the spike explained away.

The second branch is worth doing mechanically rather than by intuition, because the eye goes to whichever dimension somebody happens to chart first. A short script that computes the retry rate before and after the step change across every value of every categorical dimension — tenant, plan tier, surface, client version, locale, request class, model route — and ranks them by how much of the total change each slice accounts for, takes about twenty minutes to write and answers the question in one pass. A spike that is genuinely global shows a similar delta in nearly every slice; a spike concentrated in client_version = 3.14.0 is an instrumentation or interface story, and a spike concentrated in request_class = synthesis is a mix story. On this assistant the sweep showed the delta present in every slice at roughly the same magnitude, which killed the tenant and client theories in the first hour and is the kind of negative result that saves a day.

"Nothing was released" deserves particular suspicion, because it usually means nothing was released by this team. Provider-side model updates land without a deploy on your side. A nightly index rebuild that picked up a corpus change is a release. An experiment platform ramping a variant from 5% to 50% is a release. On this assistant the first hour of investigation went into the deploy log and found nothing; the second hour went into the feature-flag audit trail and found that a retrieval reranker had gone from 10% to 100% of traffic at 06:40 Tuesday, which was itself a red herring, because the reranker also happened to add 4.1 seconds to p95.

Flat thumbs-down is weaker evidence than it looks#

The instinct in the room was that a real quality regression would have shown up in complaints, and since complaints were flat, quality was fine. That inference is worth doing arithmetic on rather than asserting. Weekly thumbs-down volume is about 1,200. Treating the count as Poisson — the distribution that describes independent events arriving at some average rate, whose standard deviation is the square root of the count — gives √1200 ≈ 34.6, so ordinary week-to-week variation covers roughly ±69 ratings, which is ±5.8% of the level. A regression has to add more than about 70 thumbs-down a week before it is visible above that band, and that is before accounting for the fact that the level also moves with the mix of customers and the day-of-week pattern, which widen the band further.

Now ask what regression would clear that bar. A regression affecting all traffic and raising the defect rate by a third would roughly add 400 ratings, which is unmissable. A regression confined to 4% of traffic — one document type, one language, one retrieval path — that doubles the defect rate inside that slice adds about 48 ratings, which sits comfortably inside the noise. So the flat thumbs-down number rules out a broad, large regression and says nothing at all about a narrow one, and narrow is what most real regressions are.

The second reason the flat line is weak is that the two signals were not independent that week. If latency rose, a fraction of users abandoned the response entirely rather than sitting through it, and abandoned responses do not get rated. A regression can therefore hold the rating count flat while making things worse, because the population that would have complained left instead. Any argument of the form "signal B did not move, so signal A is spurious" needs the two signals to be affected by the cause in different directions, and here they were not.

Once latency explains it#

Suppose the investigation lands on the fifth branch: p95 went from 7.1s to 11.2s at 06:40 Tuesday, matching the retry step change to within ten minutes. The question is what the extra retries are now worth as a feedback signal.

The first thing they are worth is a coefficient. Fitting retry as a function of latency and the obvious confounds turns an ambiguous rate into a decomposition.

import statsmodels.formula.api as smf

# One row per response: retried (0/1), latency in seconds, plus confounds.
model = smf.logit(
    "retried ~ latency_s + C(surface) + C(tenant_tier) + C(request_class)",
    data=responses,
).fit()

print(model.params["latency_s"])   # 0.175  -> odds ratio 1.19 per second

A logistic regression states its coefficients in log-odds, which become readable once exponentiated into an odds ratio: the odds of an event are its probability divided by one minus its probability, and an odds ratio of 1.19 means that each extra second multiplies those odds by 1.19 with the other terms held where they are.

At a baseline retry rate of 4.1%, the odds are 0.0428. Four extra seconds multiplies them by 1.19⁴ ≈ 2.00, giving odds of 0.0856 and a predicted rate of 7.9%. Observed is 12.3%, so latency accounts for about half of the movement and 4.4 points remain unexplained. That residual is the part worth investigating, and it is a far better target than the raw eight points, because it is what is left after the mechanism you already understand has been subtracted.

The second thing the extra retries are worth is a negative control. There is now a large, dated, clearly-labelled population of retries that are known to be latency-driven rather than quality-driven, and any classifier or heuristic that later decides which retries indicate a defect can be tested against it. A rule that flags 60% of Tuesday's extra retries as quality failures is a rule that has been shown to be wrong on a set where the ground truth is known.

The third thing is a change to the dashboard. A retry-rate line with no latency line next to it will be misread every time this happens, and it happens several times a year. Plotting the residual — observed retry rate minus the rate predicted from latency, surface and request mix — costs one scheduled job and removes the entire class of Tuesday-morning incidents from the operational load.

The residual has to be computed against a model that is refit on a schedule, because the coefficients are not stable. The latency coefficient in particular depends on what users are doing while they wait: it was 0.175 per second in March and 0.11 in June, after a streaming change made the first tokens appear in under a second, which lowered the perceived cost of waiting even though total time to completion barely moved. A residual dashboard running on a coefficient fitted eighteen months ago will drift into being wrong in a direction nobody notices, so the refit is a monthly job whose output includes the coefficient history, and a large jump in a coefficient is itself worth an alert.

What the extra retries are not worth is evaluation cases. Converting 24,000 retried requests into the golden set on the theory that retrying means failure would fill it with requests on which the model did exactly the right thing and the user waited eleven seconds for it. Those cases are then very hard to remove, because they came from real users and everybody knows real user data is precious, so they get defended in review after review while quietly making every future comparison worse. The rule that keeps this from happening is that a behavioural signal is a candidate generator and never a label: a retry says look here, and something else has to say whether what is here is wrong.

Where the effort framing runs out#

Treating retries as effort explains the false positives, and it does not solve the harder problem, which is the false negatives. Users who give up are the group whose experience was worst, and abandonment — closing the tab, switching to search, going back to the old workflow — leaves a trace that is almost uninterpretable. A session that ends after one response might be a user whose question was answered perfectly in a single turn, or a user who read one hallucinated paragraph and never came back. Absent a downstream outcome to join against, like whether the ticket the assistant drafted a reply for was reopened, the two are indistinguishable in the log, and no amount of covariate adjustment — controlling for the other measured quantities that move alongside the one you care about, as the regression above controls for surface and request class — separates them.

The state of the art here is joining behavioural signals to task outcomes rather than modelling the behaviour alone, and the research on in-conversation feedback is worth reading if you want a sense of what these traces actually predict; User Feedback in Human-LLM Dialogues is the honest version of what implicit signals do and do not carry. On the practical side the answer is usually to instrument one real outcome per surface — ticket reopened, document accepted, purchase completed — and treat retries and edits as things that point at candidates for review rather than as the metric itself.

All three chapters so far end at the same place. Explicit ratings point at failures without measuring a rate, edits localise disagreement without labelling it, and retries mark effort without diagnosing it. Every one of them produces a pile of candidates, and the pile is large: on a normal day this assistant generates around 900 thumbs-down alone. What happens to those 900 is the difference between an evaluation suite that maps the system's behaviour and one that records who complained loudest.