Iteration caps and regression gates

Nobody deploys the final checkpoint from a training run, a checkpoint being the model's weights as saved at some point during training. You keep the checkpoint with the best validation score and discard the ones after it, because a training loop is not monotone and the last step is not privileged. A critique loop has exactly the same property and is almost always written as though it does not, with the round counter serving as both the stopping rule and, implicitly, the selection rule.

Splitting those apart is the subject of this chapter. It costs about forty lines of code, it is independent of where a loop's accuracy peaks, and on a policy-answering assistant measured over 240 cases it was worth more than the critique rounds themselves.

The checkpoint comparison is a comparison rather than an identity, and the point where it stops holding is where the rest of the work lives. Checkpoint selection is easy because there is a held-out validation set and a fixed metric computed on it, so "best so far" is a number somebody can look up. A critique loop has neither. Its rounds are produced per request, at serving time, against a question nobody has labelled, and the only thing available for comparing two answers is whatever can be computed on the spot. The selection rule itself is the easy half — keep the best — and every difficulty is in what "best" is permitted to mean when the ground truth is not in the room.

A loop needs an acceptance rule, not just a stopping rule#

Here is what "return the last thing produced" looks like when it goes wrong. The loop ran three rounds of critique and revision on questions about an employee handbook:

Q: Can an employee carry unused parental leave into the next calendar year?

ROUND 1  Not under the current policy. Parental leave entitlement lapses on
         31 December and does not carry over. (CORRECT)

ROUND 3  Unused parental leave does not carry over as a general rule, though
         the policy permits carry-over of up to 5 days where leave was
         refused for operational reasons, subject to manager approval.
         (WRONG — that provision applies to annual leave, not parental leave)

The system had the correct answer at round 1 and returned the round-3 elaboration, because the loop's contract was to return the last thing produced. No component in it was ever asked whether the last thing was better than the thing before it.

Why round 3 produced that particular answer is worth following link by link, because the same chain runs in every loop that degrades. A critique round is asked to find something to say about an answer. When the answer is already correct there is nothing substantive to find, and the round has no option of returning nothing — so it returns the most plausible criticism available, which for a correct and terse answer is that it is incomplete. The revision step then acts on that criticism, and the only way to act on "incomplete" is to add material. The material has to come from somewhere. What is nearest to hand is the model's general sense of what handbook policies contain, which is where a real carry-over provision belonging to a different leave type comes from. Each link is checkable on its own: a round that must speak, a criticism that must be actionable, a revision that can only add, and a source of additions that is not the retrieved section.

The two pieces that replace that contract are independent. The iteration cap bounds cost and bounds the degrading region established in chapter 4, on the accuracy of successive reflection rounds. The gate decides, at each round, whether the new answer replaces the incumbent, and it is the piece that turns the loop from a random walk into something that can only move in one direction. It reads two kinds of signal from the objective checks of the previous chapter: hard checks, which are pass-or-fail predicates such as schema validity or a citation that resolves, and a soft score, a continuous number between 0 and 1 for a quality that has no clean threshold.

from dataclasses import dataclass

@dataclass(frozen=True)
class Verdict:
    hard: dict[str, bool]   # check name -> passed
    soft: float             # 0..1 from a scored check, e.g. citation coverage

def beats(new: Verdict, cur: Verdict, margin: float = 0.05) -> bool:
    # 1. never accept a revision that breaks a check the incumbent passed
    for name, passed in cur.hard.items():
        if passed and not new.hard.get(name, False):
            return False
    # 2. accept immediately if it repairs a hard failure and breaks nothing
    if sum(new.hard.values()) > sum(cur.hard.values()):
        return True
    # 3. otherwise it must win on the soft score by a margin
    return new.soft >= cur.soft + margin


def gated_loop(question: str, max_rounds: int = 3) -> str:
    best = generate(question)
    best_v = score(question, best)
    rejections = 0

    for _ in range(max_rounds - 1):
        if all(best_v.hard.values()) and best_v.soft >= 0.95:
            break                      # nothing left for a round to fix
        candidate = revise(question, best, critique(question, best))
        cand_v = score(question, candidate)
        if beats(cand_v, best_v):
            best, best_v, rejections = candidate, cand_v, 0
        else:
            rejections += 1
            if rejections == 2:        # two rounds in a row produced nothing
                break
    return best

Three properties of that function do the work. The incumbent is retained on ties, so a revision has to earn its place and a round that produces a lateral rewrite costs tokens and changes nothing. A revision that breaks any check the incumbent passed is rejected outright regardless of what else it improved, which is the rule that catches the parental-leave case, where the round-3 answer cited a provision the retrieved section does not contain. And the loop exits early when the incumbent already passes everything, which on the policy assistant's traffic meant 58% of questions never ran a second round at all.

Running the parental-leave case through beats shows the three numbered clauses firing in order. The round-3 answer arrives as new and the round-1 answer is the cur incumbent. The first clause walks the incumbent's hard checks and reaches one of the policy assistant's own: that every asserted policy rule quotes a sentence from the retrieved handbook section. Round 1 passes it and round 3 does not, because the carry-over provision it cites is not in the retrieved section. The function returns False at that point and never evaluates the other two clauses. That the round-3 answer is longer, better hedged and more fluent gets no vote at all, because no clause reads any of those properties.

Which answer a candidate is compared against is the other half of the gate's design, and the weaker version of it is easy to write by accident. beats is called with best_v rather than with the verdict of the round just before, and best is reassigned only when a candidate wins. Every round is therefore measured against the best answer seen so far, not against the most recent one. The difference shows up when degradation is gradual: a gate comparing each round with the round before it will wave through a sequence of revisions that each clear the margin against their immediate predecessor while the answer walks steadily away from the best version the loop ever held. Anchoring on the incumbent makes that impossible, because the anchor only ever moves in the improving direction.

The margin parameter is the one knob worth tuning deliberately, and it should be set from the noise floor of the soft score rather than from taste. Scoring the same answer twice under a heuristic that involves any sampling gives a spread; scoring two paraphrases of the same correct answer gives a wider one. On the policy assistant, citation coverage varied by about 0.03 between paraphrases of an identical-in-substance answer, so a margin of 0.05 means a revision has to move the score by more than the metric moves for free. A margin of zero turns the gate into a coin flip on every tie, which is the naive loop again with more machinery.

Why the second number has to exceed the first is the step worth doing rather than accepting. The spread between paraphrases is movement the metric produces with no change in quality whatever, so it is available for free on every round. A margin set at that same level would therefore admit revisions whose entire advantage is noise, and would admit them at whatever rate the noise happens to fall in the candidate's favour. A margin a couple of hundredths above it requires a candidate to clear a distance the metric cannot manufacture on its own, so a revision that passes has at least a claim to being different rather than merely re-rolled. Raising it further costs something symmetric and worth naming: real improvements smaller than the margin are rejected along with the noise, which is a cost this chapter counts later, in cases rather than in points.

The scoring function is the same set of objective signals from the previous chapter, used a third way: as a comparator between two candidate answers rather than as a pass-or-fail on one. For the policy assistant the hard checks were that every asserted policy rule quoted a sentence from the retrieved handbook section, that no answer named a policy document not in the retrieval set, and that the answer addressed the question's leave type. The soft score was the fraction of sentences carrying a supported citation.

What the gate is worth#

Rerunning the same 240-case eval across four configurations, three runs each, gives the numbers that decide the design.

The gate is worth 6.7 points on its own, and it is worth them entirely by not losing answers the loop already had. The 23 cases where a correct round-1 answer was discarded were not a tail; they were 9.6% of the eval, and recovering them moves the headline more than the critique rounds did.

Both of those figures come out of the table by subtraction and division, and doing them in the open is what keeps the gate's contribution from being confused with the loop's. The two rows that differ only in the gate are the second and the third: same three rounds, same critique prompt, one returning the last answer and one returning the best so far. Their accuracies are 76.2% and 82.9%, and the distance between them is the 6.7 points. The amount of generation did not move across those two rows — mean output tokens read 940 against 910, so the gated run produced slightly less text rather than more — which rules out the explanation that the gain came from extra work. The 9.6% is the 23 discarded cases divided by the 240 in the eval, and the conversion is worth making because a count of 23 sounds like a tail while a rate of nearly one in ten does not.

The last row is the cheapest of the good configurations. Given a round-2 peak, the third round buys 0.8 points for 390 extra output tokens per answer and 3.6 seconds of p95, and the gate rejected 79% of the round-3 revisions it was offered, which is a direct statement that round 3 mostly produces work to be thrown away.

Each of those three numbers is a difference between the last two rows, and each is a subtraction anyone can repeat from the table. Accuracy runs 82.1% to 82.9%, which is the 0.8 points. Mean output tokens run 520 to 910, which is the 390. The p95 runs 4.4 seconds to 8.0, which is the 3.6. Setting the three beside one another is what converts a preference into a decision: the third round comes close to doubling both the output tokens and the tail latency in order to move accuracy by less than a point, on an eval whose run-to-run spread is not zero. The 79% rejection rate says the same thing from the other end, since most of what round 3 produces is never returned to anyone.

Three fields per request make all of this legible afterwards, and they cost nothing to emit: the round whose answer was returned, the number of revisions offered, and the number rejected with the reason. A month of that data answers questions that otherwise require a special investigation — which question types ever benefit from a second round, whether the early exit is firing on the traffic you expected, whether a particular hard check is doing all the rejecting. On the policy assistant, the returned-round field showed that 71% of answers came from round 1, 27% from round 2 and 2% from round 3, which is the same finding as a per-round sweep arriving continuously and for free from production rather than from an eval that has to be commissioned.

Those returned-round shares also cross-check the early exit, which is a consistency worth confirming rather than assuming. The exit fires when the incumbent already passes everything, and on this traffic that was 58% of questions, every one of which returns a round-1 answer. Round 1 supplied 71% of answers in total. The gap between those two shares — a further tenth or so of questions — ran a second round, was offered a revision, and had it rejected. That population is where the wasted tokens sit: a round was generated, scored and thrown away, and it is only countable because the rejection count is emitted.

That rejection rate is worth logging permanently. It is the cheapest available signal for whether a round still earns its place: when the gate starts rejecting most of what a round produces, the round is a cost line rather than a quality mechanism, and when a model upgrade changes the answer, the rejection rate moves before the accuracy does.

Acting on a sweep that names round 2 as the peak is not merely setting max_rounds = 2, and treating it as a one-line change is how a team removes the symptom and keeps the fault. A cap deletes the rounds after the peak and does nothing whatever about a round inside the cap that hands back something worse than what it was given. The two changes fix different things and are both needed. On a documentation-QA agent whose later rounds have a flag-invention problem — answers acquiring command-line flags the source documentation does not contain — the cap is what stops answers from growing, because rounds 3 onward no longer exist, and the gate with an entity check among its hard checks is what stops the assertive elaborations being returned, since a revision naming a flag absent from the source is rejected and the earlier, hedged, correct answer is kept. Cap without gate leaves round 2 free to invent. Gate without cap leaves the tokens being spent on rounds whose output is nearly always discarded.

Comparing two answers when the checks tie#

The gate above assumes something can distinguish the candidates. Often nothing does — both answers pass every hard check and the soft scores are 0.91 and 0.93, which is inside the noise of a citation-coverage heuristic. The margin in beats handles that by defaulting to the incumbent, and defaulting to the incumbent is the correct bias, since the incumbent has survived at least one round of scrutiny and the candidate has not.

The tempting alternative is to have a model choose between them, and it can be made to work with care. Left alone it does not: presenting the incumbent and the candidate to a same-family judge and asking which is better returned the second-presented answer 61% of the time on a set where the two answers had been randomly assigned to positions, which is position bias of a size that swamps any quality signal. Requiring the judge to prefer the same answer when the order is swapped removes it, at the cost of running the comparison twice and finding that 22% of pairs are indecisive, which then default to the incumbent.

def model_prefers_candidate(question, incumbent, candidate) -> bool:
    a = judge(question, first=incumbent, second=candidate)   # -> "first" | "second"
    b = judge(question, first=candidate, second=incumbent)
    return a == "second" and b == "first"   # must agree in both orders

That is a rung-two or rung-three signal doing a job the objective checks could not, and its recall and false-alarm rate need measuring like anything else. Its value here is bounded by what it defaults to when it cannot decide, which is why the both-orders requirement matters more than the judge prompt.

The gate inherits the quality of its scorer#

A gate is a comparator wrapped around a scorer, so it is worth exactly what the scorer is worth and no more. Gating on an unaided same-model judgement of which answer is better reproduces the problem from chapter 1, on unaided self-critique, with extra steps and extra latency, and it will systematically prefer the longer, more confident, more elaborated answer — which is precisely the answer that the later rounds produce and the one you were trying to reject. A gate whose scorer has no objective component is a mechanism for laundering a preference for verbosity into an acceptance decision.

There are tasks where no objective component exists at all — rewrite this paragraph in a warmer register, make this summary more useful to a sales team — and the honest position on those is that the loop should not run. There is nothing to gate on, so a multi-round loop is a sampler that will drift toward whatever the model's preference gradient points at, which is length and confidence. What can be done instead is to generate two or three independent candidates at the same round and pick among them with a human, a preference model trained on your own labels, or a rule you are willing to write down, since choosing among siblings does not accumulate justifications the way choosing between an answer and its own revision does. That is a different pattern from critique-and-revise and it does not inherit the failure mode.

The gate is also conservative by construction, which has a cost. It cannot rescue a wrong round-1 answer that passes all the checks, since a correct revision of it will tie on the hard checks and fall inside the soft margin, and the incumbent wins ties. On the policy assistant that showed up as 4 cases in 240 where the loop found a better answer at round 2 and the gate discarded it. Trading 4 recovered errors against 23 preserved correct answers was the right trade there, and the ratio is a property of that check suite rather than a general law — a system with weak checks and a strong revision step could reasonably run a smaller margin.

The chain that produces those four cases is short, and following it shows the blind spot is structural rather than a tuning error. A wrong round-1 answer that passes every hard check leaves the second clause of beats with nothing to fire on, because the count of passing checks cannot rise above a count that is already complete. The decision therefore falls to the third clause, which asks whether the candidate wins on the soft score by the margin. The soft score here is the fraction of sentences carrying a supported citation, and a wrong answer can be fully cited — it quotes real sentences from the retrieved section and applies them to the wrong leave type. The correct revision scores about the same, lands inside the margin, and loses. The gate cannot see the error because nothing it reads is sensitive to that error, which is why the move that addresses it is a check that is sensitive to it. Lowering the margin admits noise on every request in order to rescue a case the gate still cannot identify.

The cap and the gate are worth separating because they answer different questions. A cap set from a per-round sweep keeps the loop out of the region where it reliably degrades, and it is a number that expires. A gate keeps every round honest inside that region, whatever the sweep said, by requiring each revision to beat the answer it replaced on signals that were not produced by the model doing the replacing. Of the two, the gate is the one that moved the headline here, and it did so entirely by not losing ground the loop had already taken.

There is one thing it does nothing about at all. Every configuration in that table still runs a critique round on every request that does not exit early, and the p95 for the shipped configuration is more than double the single-pass baseline. The gate improved what the loop returns without touching how many requests enter it, and for most features that second question is where the money is.