A flat mean hides a loop that helps half and harms half

The three-pass loop had been live for a month. The weekly quality report showed 73.9 against 73.6 for the single-pass version it replaced, inside the noise floor, and because nothing looked wrong nobody opened it after week two. In the same month, support escalated four tickets with the same complaint: the answer told the merchant one thing in the second paragraph and the opposite in the fourth. One told a merchant that a disputed transaction had already been debited and then, two paragraphs later, that no funds would move until the dispute closed. The escalations were filed against the drafting system as a whole, tagged as prompt bugs, and routed to whoever was on the rota, and none of the four mentioned the loop because there was no reason to think of it.

A flat mean over a loop tells you the sum of what it did is zero. It does not tell you the loop did nothing, and the difference between those two readings is the whole month. The instrumented rerun of that week's 200 sampled tickets, comparing the looped answer against the single-pass answer for the same ticket, split like this:

Getting that table at all required a change to how the weekly report was produced. The original report scored 200 looped answers this week and compared the mean against 200 single-pass answers scored last month, which is an unpaired comparison between two different sets of tickets, and an unpaired comparison can only ever produce an aggregate. A paired design runs the same ticket through both arms, an arm being one of the two configurations under comparison, and subtracts, so the unit of measurement becomes a delta on one ticket rather than a difference of means. It costs one extra generation call per sampled ticket — around $0.42 for a weekly sample of 200 — and it is what turns a single ambiguous number into the distribution above.

The reason an unpaired design cannot be rescued by better statistics is worth walking, because teams try. Under pairing, each ticket yields one number: this ticket's looped score minus this ticket's single-pass score. That number is a property of the ticket, so it can be counted, sorted, bucketed by sign and grouped by slice. Without pairing, no ticket has a difference attached to it, because no ticket was answered both ways. All that exists is one mean from one set of tickets and another mean from a different set, and subtracting them produces exactly one number for the whole week. There is nothing to bucket by sign, because there are no signs — the subtraction happened after the averaging, and averaging is where the signs went.

Those columns sum to +0.5, which is the flat weekly number. A loop that does nothing at all also produces +0.5, and so does a loop that improves every case by half a point. Three qualitatively different systems, one aggregate, and the aggregate is the only thing anyone was looking at.

Assembling that sum from the table shows how little the aggregate is carrying. Forty-one hundredths of a gain of 6.8 points contributes a little under three points. Thirty-seven hundredths of a loss of 6.1 points takes back a little over two and a quarter. The middle group contributes almost nothing, since twenty-two hundredths of two tenths of a point is a rounding error. The residue is the half point the report was showing, and it is a residue in the strict sense: what is left after two much larger quantities of opposite sign have very nearly cancelled. One further caution about reading that residue against the weekly report's three-tenths-of-a-point gap. The two are not the same quantity — the residue is a paired mean over 200 tickets answered both ways, and the report's gap is a difference between two unpaired means from two different weeks. That they land in the same neighbourhood is unsurprising and is not a confirmation of anything.

Instrument the sign of the delta, not the size of the mean#

The fix is to log per case and per iteration and to aggregate only after the sign has been recorded. Every pass of every request should emit a row, and the row is small.

{
  "run_id": "8c1f...",
  "ticket_id": "T-40219",
  "slice": "payouts",
  "pass": 3,
  "score": 0.61,
  "prev_score": 0.68,
  "delta": -0.07,
  "gate": "reject",
  "gate_source": "retrieval",
  "findings_emitted": 3,
  "findings_resolved": 1,
  "findings_reraised": 2,
  "chars_changed": 412,
  "paragraphs_touched": [2],
  "calls": 2,
  "cost_usd": 0.0041
}

Read the arithmetic inside that row before using the field name, because the word delta is doing two jobs in this chapter and they are not the same measurement. Here score minus prev_score is 0.61 minus 0.68, which is the −0.07 recorded, so this row's delta is within the loop: this pass against the previous pass of the same run. The delta in the table above is between arms: the looped answer against the single-pass answer for the same ticket. A run can post positive within-loop deltas at every pass and still come out below its own single-pass answer, because the two quantities have different baselines. Keeping them straight is the difference between a chart that says the loop is improving drafts and one that says the loop is improving on itself.

Three aggregations over that table answer the questions the mean cannot. The first is the improvement and regression rates as separate series, so the 41% and the 37% are two lines on a chart rather than one number that averages them; a week where both rise is a loop becoming more aggressive, and a week where the regression line rises alone is an incident. The second is gate behaviour by slice, a slice being a category of request the system routes and reports on together, such as payout questions or disputes: acceptance rate, rejection rate and abstention rate, because a slice where the gate abstains on every request is a slice where the loop is running ungated, and that is exactly where regressions concentrate. The third is the reversion signal from chapter 1 — findings re-raised per pass — which climbs before quality falls, because a reviser producing motion rather than repair is the leading indicator of a loop that has run out of real findings.

SELECT slice,
       countIf(delta >  0.03) AS improved,
       countIf(delta < -0.03) AS regressed,
       countIf(gate = 'abstain') AS ungated,
       round(avg(delta), 3)   AS mean_delta
FROM loop_passes
WHERE day >= today() - 7
GROUP BY slice
ORDER BY regressed DESC;

Note what that query is counting, because the table it reads has one row per pass rather than one row per ticket. A three-pass run contributes three rows, so improved and regressed are counts of passes and a ticket that improved twice and regressed once appears in both columns. Turning those into the per-ticket rates the previous section talked about needs a second grouping, by run_id, before the counting. The distinction is easy to lose because both readings produce a plausible number and neither errors, and it decides what "a 58% regression rate" means: more than half the passes, or more than half the tickets. Those are different claims about different things, and a slice with a high pass count can show a high pass-level rate while most of its tickets are fine.

Running that query against the month of data put the whole story on one screen. The payouts and disputes slices had retrieval snippets on nearly every ticket, gate acceptance around 34%, and improvement rates near 60% against regression rates near 12%. The how-do-I slice — phrasing help, general guidance, anything without a policy lookup — had an abstention rate of 96%, which meant the gate was keeping the incumbent by default and the passes were running anyway, and the improvement and regression rates there were 24% and 58%. The uncovered slice was 44% of traffic and it was carrying the entire negative half of the distribution.

Put a floor under what fixing that slice is worth, because the arithmetic decides whether the change is a tidy-up or the whole answer. The uncovered slice is 44% of traffic and regresses at 58%, so regressions arising there account for roughly a quarter of all requests. The overall regression rate is 37%. So something close to two thirds of every regression the system produces comes from one named slice, and routing that slice away from the loop removes them without touching anything else. That is a floor rather than an estimate, in both directions. It is a floor on the benefit because the slice's remaining regressions are the ones a fix cannot reach, and it is only a floor because the figures come from two different samples — a month of query output and one week's 200 paired tickets — so the two thirds is an order of magnitude and not an account. The same arithmetic also prices what the fix costs: that slice's 24% improvement rate goes with it.

Disagreement between the loop's own components is the fourth thing to record, and it is free because the components already emit it. A critic that raises a finding the reviser rejects with a reason, a revision the gate refuses on a scored signal, and a finding the critic re-raises after the reviser marked it applied are three different disagreements, and their rates over time describe the loop's internal state better than its output score does. On the payments drafter, the how-do-I slice showed a reviser rejection rate near zero — the reviser agreed with every finding — which reads as harmony and is closer to the opposite: two components sampled from the same weights, with no external signal between them, converging on each other rather than on the answer.

None of this instrumentation is expensive relative to the loop it watches. The rows above are roughly 300 bytes each, a three-pass run emits three of them, and at 40,000 requests a day that is 36 MB a day before compression. The scoring is the part that costs, since a per-case delta needs the answer scored under both arms, which is why it runs on a sample rather than on all traffic. A 5% sample at this volume is 2,000 paired comparisons a day, enough to detect a ten-point shift in the regression rate within hours.

Both of those figures fall straight out of multiplication, and doing it once is what makes the conclusion arguable rather than asserted. Three hundred bytes times three rows is under a kilobyte per request, and under a kilobyte times forty thousand requests is the tens of megabytes quoted — a rounding error against the model bill for the same traffic. Five in a hundred of forty thousand is the two thousand paired comparisons, and the reason sampling is confined to the scoring rather than the logging is that a log row costs bytes while a paired comparison costs an extra generation call. The asymmetry is what decides the design: log everything, score a sample.

The claim about detecting a ten-point shift within hours rests on the same sample size and is checkable. A regression rate is a proportion, and the uncertainty on a proportion shrinks with the square root of the count behind it, so a few hundred paired comparisons — a few hours of that daily sample — puts the wobble on a rate near two in five at a couple of percentage points. A ten-point move is several times that wobble, which is why it shows up quickly, and a two-point move is inside it, which is why it does not show up at all. The practical consequence is that this instrument is built to catch a step change and is not built to catch a slow drift. Catching drift needs the same series read over weeks rather than hours, and the stop conditions below are written against step changes for exactly that reason.

Confirming the loop is what is doing it#

Correlation between a slice and a regression rate is not yet a cause, and the cheapest confirmation is a replay: take the four escalated tickets and the 74 regressed cases from the sample, run each through the single-pass configuration, and score both. If the single-pass answers do not contain the defect, the loop is producing it rather than passing it through. All four escalations came back clean on a single pass.

That count of 74 is the table at the top read back the other way, and checking it is worth two seconds: thirty-seven hundredths of the 200 sampled tickets is the 74 regressed cases, so the replay set is exactly the negative group and not a selection from it. Replaying the whole group rather than a sample of it matters here, because the question being asked is whether any of the regressions survive a single pass, and a sample can only ever bound that.

The second confirmation is provenance, meaning a record of which pass produced which piece of the final text, which is available because chapter 1's findings carry locations and chapter 4's ledger keeps every pass. For each of the four contradictions, the question is whether the contradicting paragraphs were touched in the same pass. In all four, they were not: paragraph 2 had been rewritten at pass 2 under a finding about clarity, and paragraph 4 was byte-identical to the original draft.

Where the instrumentation is blind#

The instrumentation has the same blind spot as everything it is built on, which is worth being explicit about before anyone treats a green dashboard as an assurance. Every delta in this chapter is a difference of scores from the proxies of chapter 3, so a regression the rubric and the citation check both consider fine leaves no trace in any of these series. The four escalations were found by support rather than by the monitoring, and after these changes the monitoring would catch that specific class because a consistency check now exists for it. The general form of the problem — a failure mode nobody has written a check for is invisible until a human complains — does not go away, which is the argument for keeping the sampled human review from chapter 3 running even when the charts are flat.

The consistency check's own numbers make the same point from inside the fix. It caught 31 of the 74 regressed cases, which is a little under half of the group the paired scoring had already identified, and that group is itself only the regressions the proxies can see. So one cheap check addresses under half of the visible half, and the honest way to report it is as a named class of defect now covered rather than as a proportion of anything. A proportion needs a denominator, and the denominator here is the number of ways this system can produce a worse answer, which nobody knows.

The stop conditions follow from the same table. A regression rate above 25% on any slice over a rolling seven days disables the loop for that slice automatically and pages whoever owns it; an abstention rate above 80% on a slice raises a ticket, because a slice the gate cannot score should not be running a gated loop; and a re-raise rate above 1.5 findings per pass caps that slice at two passes. Each of those thresholds came from the distribution the instrumentation produced rather than from judgement, which is the argument for gathering it before you need it.

Three thresholds, three different jobs, and the ordering between them is deliberate. The regression rate is the outcome measure, so it gets the strongest action and the longest window, because it is the slowest to move and the most expensive to act on wrongly. The abstention rate is a structural fact about the gate rather than a quality signal, so it raises a ticket for a person instead of tripping anything, since the right response is a design decision about coverage. The re-raise rate is the leading indicator from chapter 1, so its action is the mildest — a cap rather than a disable — because it fires before any damage has been measured and is the most likely of the three to be a false alarm.

The system now improves what it can measure, stops where it cannot, and reports honestly on both. The remaining question is whether the loop is the best thing to do with the calls it consumes, because three passes over one draft is not the only way to spend three model calls on a request.