Eighty percent of raters is not eighty percent of answers

The slide had two numbers on it and one sentence underneath. Last month the assistant embedded in a B2B support console produced 2,140,000 responses. Of those, 7,900 received a thumbs rating, and 6,320 of the ratings were thumbs-down. The sentence underneath read: four out of five answers are bad. By the end of the review the head of product had asked for a pause on the roadmap, an engineer had been assigned to "fix quality", and nobody in the room had said the thing that made the whole slide unusable, which is that 7,900 is 0.37% of 2,140,000 and the people who supplied those 7,900 opinions did not arrive at random.

That slide is the visible end of a machine the product had been running for a year. A production feedback loop is the path a signal takes from a real user back into the system: something the user does is captured against the response that prompted it, somebody works out what it means, a change is made on the strength of it, and a later measurement says whether the change helped anyone. It exists because the test set written before launch was assembled by the people who built the product and therefore holds the requests they thought of, while the traffic holds the ones nobody thought of — the ticket pasted in whole with three levels of quoting, the question whose premise is false, the customer who writes in Portuguese and expects the answer in English. Production is the only place those arrive, which makes it the only place the next round of test cases can come from.

The signals arrive in two kinds, and the difference between them organises the rest of this topic. Explicit feedback is a judgement the user deliberately states: the thumbs pair under an answer, a star rating, a sentence typed into a what went wrong? box. Implicit feedback is behaviour the product was recording anyway and from which a judgement has to be inferred: the user rewrites the draft before sending it, copies two paragraphs out of five, presses regenerate, or closes the tab having done none of those. Both are familiar from the user's side, since the thumbs under a chatbot reply and the regenerate arrow beside it are the same controls this chapter is about; on the engineering side they arrive as rows joined to the request and the response that produced them. The work of the loop is deciding which of those rows deserve to become evaluation cases — a stored request, the context it ran against, and an agreed statement of what the answer should have been, replayed against every later version of the system — and the slide above is what happens when the first kind of signal is instead read as a measurement of the system rather than of the people who supplied it.

Clicking a thumb costs something. It costs the second it takes to move the cursor, the small social awkwardness of registering a complaint, and the interruption to whatever the person was actually doing, which was closing a support ticket rather than evaluating a language model. People pay that cost when they have a reason, and dissatisfaction is a much stronger reason than satisfaction. An answer that was fine gets read and used, and the user moves on; an answer that invented a refund policy gets a thumbs-down because the user wants the record to show that this happened. The ratio between the two kinds of rating is therefore a fact about who bothered to rate, and the population of people who bothered is defined by exactly the property being measured.

The same two numbers survive almost any true defect rate#

The way to see how little the 80% constrains is to hold the observed numbers fixed and ask what defect rate could have produced them. Write the rating rate as 0.4% of all responses, of which four fifths are negative, so negative ratings are 0.32% of responses and positive ratings are 0.08%. If the true fraction of defective responses is d, then the probability that a dissatisfied user rates is 0.0032 / d, and the probability that a satisfied user rates is 0.0008 / (1 − d). Every row below reproduces the slide exactly.

Reading the slide as "four in five answers are bad" is the bottom row, and the bottom row says that a satisfied user and a dissatisfied user are equally likely to reach for the control. Nobody who has watched a user session believes that. The rows people find plausible sit in the middle of the table, where dissatisfied users rate somewhere between fifteen and seventy times more often than satisfied ones, and those rows put the true defect rate between 2% and 20%. That is a range spanning an order of magnitude, which is the honest summary of what 7,900 self-selected opinions tell you about 2,140,000 responses.

The obvious response to a 0.4% rating rate is to collect more ratings, and it does not work the way people hope. Making the control larger, adding a prompt after long conversations, or gating the next action behind a rating raises the rate — a modal that asks was this helpful? before dismissing can reach 8-12% — and it changes the population in a new direction rather than removing the selection. Users who answer a modal to make it go away click whichever button is nearest, users who are in a hurry click the positive one because it looks like the dismiss action, and the resulting data is biased toward whatever the interface makes cheap. Response-rate interventions move the bias; they do not convert a self-selected sample into a random one, and the only thing that does is drawing the sample yourself.

The propensity ratio is not a constant of nature either. It moves with the interface — a thumbs pair rendered inline under every message collects a different population from one hidden behind an overflow menu — and it moves with the user's relationship to the product. Internal staff dogfooding a tool rate at ten or twenty times the rate of external customers, because they have been asked to, and their ratings will dominate any month in which a new team is onboarded. Enterprise customers with a named customer success manager rate more than self-serve ones, because they have learned that complaints get acted on. None of these movements is a change in quality, and all of them change the headline ratio.

Getting a real number means paying for it once#

A rate that means something comes from a sample that was drawn without reference to anyone's satisfaction, and then labelled by people who were told what "defective" means. The rubric work here is the same discipline as building a golden dataset — the categories, the adjudication rule, the version pin — applied to a random draw from live traffic rather than to curated inputs.

The sampling itself should be deterministic so that two people running the query a week apart pull the same rows and the sample can be re-drawn after a schema migration without shifting underneath the labels.

-- Deterministic 1-in-5350 sample of one week of traffic.
-- The salt pins the sample: rerunning this query returns the same rows.
select
  r.response_id,
  r.tenant_id,
  r.request_class,
  r.created_at
from responses r
where r.created_at >= date '2026-03-02'
  and r.created_at <  date '2026-03-09'
  and mod(abs(hashtext('qsample-2026-w10:' || r.response_id::text)), 5350) = 0
order by r.created_at;

At 2.14 million responses a month, that lands about 100 rows a week and 400 a month. Two raters label each one independently against a versioned rubric, and disagreements go to a third pass rather than being averaged away.

{
  "response_id": "resp_9f31c8a2",
  "sample_id": "qsample-2026-w10",
  "rubric_version": "support-quality@4",
  "labels": [
    { "rater": "ana",   "verdict": "defect", "category": "unsupported_claim", "severity": 2 },
    { "rater": "priya", "verdict": "ok",     "category": null,               "severity": 0 }
  ],
  "adjudication": { "rater": "tomas", "verdict": "defect", "category": "unsupported_claim", "severity": 2 },
  "redaction_state": "pseudonymised"
}

Four hundred labelled responses a month, at the six or seven minutes each one takes when the labeller has to open the retrieved documents and check a claim, is about forty-five hours of work, split across three people it is a day and a half each. In exchange the organisation gets a number with a stated confidence interval: if 26 of 400 come back defective, that is 6.5%, and the 95% interval runs from roughly 4.3% to 9.4%. The interval is wide, and saying so is part of the deliverable, because a monthly figure of 6.5% ± 2.5 points is a different management object from "80% bad" — it supports a decision about whether this is a crisis, and it does not support a claim that quality improved by one point last month.

Two design choices make those 400 rows worth several times what a naive draw would be. The first is stratification by request class rather than uniform sampling, because the uniform draw is dominated by whatever the most common request type is and leaves eight rows for the class that carries the business risk. Sampling 100 per class from four classes, and then re-weighting to traffic share when computing the overall rate, produces both a global number and four usable per-class numbers from the same labelling budget. The second is measuring agreement between raters and reporting it. On the first month of this rubric the two independent raters agreed on 84% of items, which is a Cohen's kappa — the agreement rate after the agreement two raters would have reached by guessing is discounted — of about 0.55, moderate and low enough that the rubric had three categories collapsed and two examples added before the numbers were quoted to anyone. A defect rate produced by raters who agree at chance is a number with a confidence interval that nobody has computed.

That last limitation is worth being explicit about, because it is where teams misuse the sample after having built it correctly. Four hundred labels a month cannot detect small movements. To distinguish 6.5% from 5.5% with any confidence you need thousands of labels, which nobody is going to fund monthly. The sample is a level, checked quarterly, that tells you roughly where the system sits and catches a large regression. It is not a monitor, and it is not the live metric that 15.13 covered; those run continuously on automated scorers over far more traffic, and the labelled sample's job is partly to tell you whether those automated scorers are still calibrated against human judgement.

What the ratings are still for#

None of this makes the thumbs worthless, and the failure mode of a team that has just learned about selection bias is to stop looking at explicit feedback altogether. The 6,320 thumbs-down are a poor estimator of a rate and an excellent index of failures. Each one is a user pointing at a specific response and saying that this particular thing was wrong, usually within seconds of it happening, with the full trace still available. That is a pointer into the log, and pointers are what the later chapters build eval cases from.

Explicit feedback also carries the one thing implicit signals never do, which is intent. A free-text comment attached to a thumbs-down — it quoted the old SLA, we changed that in January — names the failure in a way no behavioural trace can. Comment rates are low, often 10-15% of thumbs-down, so on this product that is roughly 800 comments a month, and those 800 are the highest-information-per-byte artifact the feedback system produces.

The third use is detection of sharp changes. Selection bias is roughly stable week to week, so while the level is uninterpretable, a step change in the level is a real event. Thumbs-down volume tripling on a Thursday means something happened, even though the something might be a UI change that moved the control rather than a regression. That asymmetry — the level is meaningless and the derivative is informative — is the correct way to put explicit feedback on a dashboard.

Both of these are copies of what users wrote#

The 7,900 ratings are not just booleans. Each one is stored against a response, and the response and the request that produced it are the customer's content: the ticket text, the customer's name, the order number the agent pasted in. The 400-row labelled sample is the same material, copied a second time into a labelling tool, and read by three employees who had no other reason to see those particular tickets. Whatever retention rule governs the conversation log governs both of these copies, and a labelling spreadsheet in a shared drive with no expiry is a straightforward violation of a policy the company has already written down. Chapter 5 deals with where these artifacts live and what redaction leaves usable; for now the point is that building a feedback capability creates new copies of user data by design, and the copies need owners on day one rather than after the first audit.

There is a narrower version of this problem in the sampling query above. Sampling deterministically by response id is reproducible, which is good, and it also means the sample is not refreshed when a user exercises a deletion right — the row is gone from responses but the labels persist keyed to an id that no longer resolves. Systems that get this right keep the label store joined to a subject index rather than treating labels as free-standing facts.

The larger gap left open here is coverage. A 0.4% rating rate means 99.6% of responses generated no explicit statement of opinion at all, and a 400-row monthly sample covers 0.02% of traffic. Neither number is going to improve much: prompting harder for ratings raises the rate a little and biases it differently, and labelling ten times as much costs ten times as much. Meanwhile every one of those 2.14 million responses was followed by something the user did — they copied it, they edited it, they asked again, they closed the tab — and those actions are recorded on essentially all of them. The next chapter starts with a product where one of those behavioural traces is available on 100% of responses and points directly at the part of the answer the user disagreed with.