Return the best draft the loop saw, not the last one

Take the curve from chapter 2 exactly as measured — 62.4, 71.0, 73.2, 72.4, 70.1 — and ask what a five-pass loop returns. It returns the pass-five draft, at 70.1, having produced a 73.2-point draft two passes earlier and thrown it away. The system generated the better answer, held it in memory, spent four model calls and 13 seconds moving past it, and handed back the worse one. Nothing in the loop was ever asked whether the new draft was better than the one it replaced, because the loop's contract was to run a fixed number of passes and return the last thing produced.

The tempting repair is to lower the cap to three, which recovers 3.1 points on the mean and is the wrong lesson. A cap is a single number chosen from an average, while the damage is per case, and the per-case picture on the 300-ticket set is not concentrated anywhere near pass three.

A cap of three returns a draft worse than one the loop already had for 21% of tickets, because those tickets never improved on their original draft and then declined. It also gives up on the 21% that were still improving at pass four or five. Whatever number you choose, roughly a fifth to a third of requests get an answer the loop had already beaten, and no choice of cap removes that, because the cap is deciding when to stop rather than what to keep. Those are two different rules and they need two different mechanisms.

Two different losses are hiding in that table, and separating them is what shows why no cap can empty both. The first loss is damage: a returned draft worse than the original the run started from. That is the 21% row read literally — the original was never beaten, so any later draft handed back is a step down from where the system began. The second loss is forgone gain: a returned draft worse than one the run itself produced and is still holding in the ledger. Under a cap of three the second loss covers the 34% that peaked at pass two as well, so between the two rows more than half the tickets end up with a better answer sitting in storage than the one handed back. Raising the cap converts tickets from the second loss into the first, and lowering it does the reverse. The distribution has weight at every pass, and one number cannot be right at five places at once.

That is the shape of the mistake worth naming, because it is easy to make while looking straight at the table. The cap is a quantity that stays fixed across every request, and the pass at which a ticket peaks is a property that varies across all five rows. Choosing the cap by moving it until the mean is highest holds the fixed quantity up against an average of the varying one, which is exactly the comparison that cannot see the tails. The mean improves and the tails stay where they are, because nothing in a cap addresses them.

The gate, the cap and the fallback are three separate things#

The acceptance gate decides whether a revision replaces the incumbent, using the external signal from chapter 3. The cap decides when to stop spending. The fallback decides what comes back when something goes wrong. Written together they are short, and the shortness is the point — most of the value in this chapter is in a dozen lines that most loops do not have.

def run(ticket, max_passes: int = 4, budget_usd: float = 0.02, deadline_s: float = 20.0):
    started = time.monotonic()
    draft = generate(ticket)
    best, best_v = draft, evaluate(ticket, draft)      # ch.3 Verdict
    ledger = Run(run_id=uuid4().hex, prompt=ticket.text)
    spent = cost_of_last_call()
    rejections = 0

    for i in range(2, max_passes + 1):
        if spent > budget_usd or time.monotonic() - started > deadline_s:
            break
        findings = critique(ticket, best)
        if not findings:
            break                                      # nothing to act on
        candidate = revise(ticket, best, findings)
        cand_v = evaluate(ticket, candidate)
        spent += cost_of_last_calls()
        ledger.passes.append(Pass(i, candidate, findings, calls=2))

        if accepts(cand_v, best_v):
            best, best_v, rejections = candidate, cand_v, 0
        else:
            rejections += 1
            if rejections == 2:                        # two passes, no progress
                break
    return best, ledger

best is the whole idea: an incumbent that only moves when the gate says the challenger wins, so the return value is an argmax over the trajectory, meaning the highest-scoring draft the run produced, rather than the last element of it. Ties go to the incumbent, which matters more than it looks, because a lateral rewrite that scores identically costs two calls and buys a different arrangement of the same content, and every extra rewrite is another chance to drop a number. The two-rejection stop is a cheap dynamic cap: when the loop has produced two consecutive revisions the gate refused, it has run out of the kind of error it can find, and the remaining passes will be the cosmetic findings chapter 2 measured.

Two properties of that loop follow from the code rather than from anybody's intention, and both are worth deriving because the rest of the chapter leans on them. The first is that the score attached to best never falls. It is set once from the first draft and thereafter only reassigned inside the branch guarded by accepts, which chapter 3 defines as requiring the challenger's score to clear the incumbent's by the margin. A quantity that starts somewhere and only ever moves upward is at least where it started, so the returned draft scores at least as well as the first one. The second is that critique and revise are both handed best rather than the last candidate, so a rejected revision is discarded outright and the next pass starts again from the incumbent. That is why two rejections in a row is meaningful: the loop has now twice attacked the same text and twice failed to improve it, which is different from failing to improve two different texts.

The budget and deadline checks exist because a cap on passes is not a cap on cost. A pathological ticket with a 4,000-token retrieval context and a critic that emits eight findings per pass can cost eight times the median request, and in a support system with a p95 latency target, which is the bound the slowest one request in twenty has to respect, the deadline is the constraint that actually binds. Checking both at the top of the loop means the system degrades by returning the best draft so far rather than by timing out, which is what "safe fallback" means in practice: every exit path, including exceptions, budget exhaustion and deadline expiry, returns best, and best is never worse than the original draft because it started there and only moved on an accepted comparison.

Where those two checks sit is a design decision rather than a requirement, and it has a consequence. They run at the top of the loop, so they stop the loop from starting a pass it cannot afford, and they cannot stop one already in flight. A pass that begins just inside the budget and turns out to be the expensive one overshoots, and the overshoot is bounded by the cost of a single pass rather than by the budget. Moving the checks inside the pass would need cancellation on the model calls, which most clients make awkward, so the usual choice is to accept the overshoot and set the budget low enough that one pass of headroom is affordable. Naming it as a choice is what stops it being discovered during an incident.

Two operational details follow from that. The loop should write each pass to the ledger before the gate runs, so a rejected candidate is still on record — the rejected drafts are where the evidence lives when someone asks why the loop is not helping a particular slice. And the gate's verdict should be logged with its source field, because a run where the score came from retrieval verification and a run where the gate abstained are different events that look identical if you only log the returned answer.

That last property is worth stating precisely. The gated loop cannot return something worse than its own first draft as measured by the gate's signal. It can still return something worse by a measure the gate does not cover, which is the coverage problem from chapter 3 arriving with consequences.

What the gate is worth on the same 300 tickets#

Four configurations, three runs each, scored by the same rubric with the same ±0.9 noise floor. The regression column counts tickets where the returned answer scored more than 3 points below the ticket's own first draft.

The mean improves by 3.6 points over the best ungated configuration, which is respectable and not the interesting column. The 10th percentile, the score that the worst tenth of tickets fall below, moves from 47 to 58 and the regression rate falls from 18% to 3%, which is the shape of a change that removes downside rather than adding upside. Support tickets, medical summaries and code review comments are all domains where the cost of one bad answer exceeds the value of one good one, and a gate is bought for the percentile rather than for the mean.

The oracle row is the ceiling, obtained by letting the gate use the rubric score with human labels, which is not available at request time. The 2.6-point gap between the real gate and the oracle is the price of the proxy, and it is the honest way to report how good your signal is: not as a classification accuracy in the abstract, but as the fraction of the achievable gain the loop actually captures. Note also that the gated loop uses fewer calls than the ungated five-pass version, because the early-exit conditions fire on most tickets — 47% of them stopped after two passes with no accepted revision.

That fraction is worth computing rather than gesturing at, since it is the number the sentence above promises. Take the best ungated configuration as the starting point, at 73.2, and the oracle as the ceiling, at 79.4: the gain a perfect scorer would deliver is a little over six points. The real gate delivers 3.6 of them. So this proxy captures a little under three fifths of what a perfect one would, and that single fraction is a better description of the signal than any accuracy figure, because it is measured in the units the decision is made in. The residual is that the oracle is only a ceiling relative to this loop, at this cap, with this critic — a better critic would raise both rows and the fraction between them could move either way.

The call column is the one place in that table where the arithmetic closes exactly, and it closes in a way that confirms the reading. A four-pass loop that never exits early makes one generate call plus a critique and a revision for each pass after the first, which is seven calls — and 7.0 is precisely what the oracle row reports. So the oracle gate accepts something on essentially every pass and never triggers the two-rejection stop, which is what an oracle would do. The real gate averages 5.6 against that same ceiling of seven. A run that stops after two rejected passes has made five calls, so a mean of 5.6 sits much nearer the early-exit figure than the ceiling, which is the arithmetic behind the 47%. What that licenses is a statement about spend: gating this loop is cheaper than not gating it at the same cap. What it does not license is reading 5.6 as typical, since it is a mean over a mixture of two behaviours with nothing in between them.

"The mean went up" is not an answer to "can it get worse"#

The five-pass and three-pass rows both beat the single-pass baseline by a wide margin, and a team looking only at that column would ship either of them and consider the question closed. A mean is a sum divided by a count, so it says what happened to the total and nothing about how the total was assembled. The three-pass row's 73.2 is compatible with every ticket improving by 10.8 points, and equally compatible with 82% of tickets improving by 16 points while 18% lose 12 — which is roughly what the regression column says actually happened.

Work that second reading through, because a claim about compatibility is only worth as much as the arithmetic behind it. Eighty-two hundredths of sixteen points is a little over thirteen points of contribution, and eighteen hundredths of twelve points is a little over two points of contribution in the other direction, and the difference between them lands within a fraction of the 10.8 the mean reports. So both distributions produce the same mean, and nothing about the mean can distinguish them. That is the general property and it is worth carrying beyond this chapter: a total is compatible with every distribution that sums to it, so a total can never be evidence about a shape. Learning the shape needs the per-case numbers, which is why the regression column had to be measured separately rather than inferred.

Whether that distinction matters depends on how the output is used, and for most systems built on a critique loop it matters a great deal. Support answers are read one at a time by one merchant each, so a merchant does not experience the mean. If the loop turns a correct answer about a 3-day review window into a confident wrong one about 5 days, the fact that 82% of other merchants got better answers is not a defence, and the cost of the bad answer is a support escalation, a refund dispute or a compliance question rather than 12 rubric points. The asymmetry is the reason the 10th percentile and the regression rate belong in the summary table next to the mean.

There is also a measurement trap in accepting a mean as evidence of safety, which is that a mean computed on a frozen offline set has been optimised against. The critic prompt, the finding taxonomy, the margin and the cap were all tuned while looking at that set, so the 300 tickets have been fitted to a degree nobody intends and nobody tracks. A gate whose value shows up as regression avoidance is partially protected from this, since a hard veto on dropped currency figures does what it does regardless of which tickets it sees, and a rubric mean has no such protection.

The failures a gate leaves in place#

It does not make the loop worth running. A gate that rejects every revision leaves you paying for critique and revision calls to return the first draft, which is strictly worse than not looping, and this is the state a gated loop settles into on a task where the model's first draft is usually fine. The acceptance rate is therefore a metric to watch in its own right: a loop accepting 5% of revisions should be switched off for that slice rather than tuned.

It does not detect the errors its signal cannot see, so the 3% regression rate above is 3% by the rubric, and the rubric was built by people who were thinking about factual accuracy, completeness, escalation path and tone. A revision that degrades something outside those four dimensions passes the gate unnoticed and appears in no column of that table.

That second limit deserves one more turn, because it is the one that survives every improvement to the gate. The gate compares two drafts on a score, and the score is a function of whatever dimensions somebody enumerated. Anything not enumerated contributes zero to both sides of the comparison, so a change that only affects an unenumerated dimension leaves the comparison exactly where it was. The gate is not failing to notice such a change; it has no representation in which the change exists. The defence is a veto list, because a veto is a rule about the artifact rather than a dimension of a score, and a rule can be added the afternoon somebody discovers the dimension.

It also does not tell you which requests it helped and which it hurt. Every number in this chapter is a mean or a rate over 300 tickets, computed offline, on a set that was frozen months ago. A loop can post a flat mean in production while improving half its cases and damaging the other half, and both the gate's acceptance rate and the aggregate quality number will look unremarkable while that happens.