Production feedback loops

Read these first

• Online evaluation
• Building a golden dataset

What you will know

• Capture explicit and implicit feedback in production

  Explicit feedback is sparse and biased; implicit signals are abundant and noisy, and both may contain sensitive user content that needs redaction and retention limits.

• Compare edits, retries, and thumbs-down signals using a stated method

  User effort often makes edits and retries more informative, but latency, confusing UX, and task difficulty can produce the same behavior without a model error.

• Turn a production failure into an eval case automatically

  The generated case needs enough request, response, context, outcome, and labeling data to reproduce the failure without leaking private data or duplicating near-identical examples.

• Close the loop from feedback to a shipped improvement

  A credible loop links the new case to a change, measures it against a fixed regression set and live metrics, and preserves rollback evidence when the result is worse.

• Show the eval set growing from real traffic

  Sampling and deduplication must prevent high-volume users or recurring incidents from distorting coverage as traffic, prompts, and product behavior change.

Chapters

1. 1

   Eighty percent of raters is not eighty percent of answers

   About 0.4% of responses receive a rating and 80% of those are negative, which the review deck turns into 'four in five answers are bad'. Rating is voluntary and costs effort, so raters are selected on dissatisfaction and the ratio describes them, not the system. Getting a real rate means drawing a random sample and labelling it — and treating both the ratings and the sample as copies of user content.

   Read chapter
2. 2

   An edit points at the span the user disagreed with

   A drafting product has thumbs on 0.3% of responses and the diff between the generated draft and what was actually sent on essentially all of them. Coverage and specificity both favour the diff: it exists everywhere and localises the disagreement. Learn to separate correction from personalisation, set a floor so trivial touch-ups do not register as defects, and decide what the sparse signal is still worth keeping for.

   Read chapter
3. 3

   Retries measure user effort, and latency raises effort

   Regenerate rate goes from 4% to 12% on a Tuesday with nothing released and thumbs-down volume flat. Retrying is an effort signal, and effort rises with p95 latency, with a button that moved in a redesign, and with a week of harder questions. Covers the order to check things in, why a flat explicit signal neither confirms nor refutes a regression, and what to do with the extra retries once latency explains them.

   Read chapter
4. 4

   Four hundred copies of one failure is weight, not coverage

   Yesterday's candidates: 900 thumbs-down, of which 400 are one enterprise customer hitting one broken document type and 120 are duplicates of an incident already fixed. Keeping all of them does not add coverage, it adds weight, and every later evaluation gets dominated by that document type. Covers what one case must carry to reproduce a failure, and the sampling and deduplication rules that keep the set a map of behaviours.

   Read chapter
5. 5

   An eval case carries the obligations of the log it came from

   The set grown from production now holds customer names, email addresses, and order numbers, and it sits in the application repository beside the code. Each case is a copy of customer data subject to the same retention and deletion rules as the log it was drawn from, and a repository — immutable history, clones everywhere — honours neither. Covers where these sets belong and what redaction still leaves usable.

   Read chapter
6. 6

   Passing the sixty cases you tuned on is not evidence

   Sixty production failures become cases, the prompt is iterated until all sixty pass, and those sixty have quietly become the objective rather than the test. Shipping needs a pre-existing regression set, untouched this round, that is no worse, and a live metric that moves after release. Covers the six-month review where the suite goes from 71% to 94% while thumbs-down and support tickets stay flat, and what that combination actually means.

   Read chapter