The same weights that wrote it are the ones judging it

The payments drafter's loop was allowed to run five passes over a frozen set of 300 tickets, each final draft scored 0–100 by a rubric, which is a fixed list of dimensions with rules for awarding points on each, covering factual accuracy, completeness, correct escalation path and tone. Rerunning the same configuration three times put the noise floor — the amount the mean moves between identical runs for no reason other than the model's own randomness — at ±0.9 points, so anything under two points of movement is not a result. The means came out at 62.4 after the first draft, 71.0 after one revision, 73.2 after two, 72.4 after three and 70.1 after four.

Everything the loop is worth arrives by pass three, and it arrives unevenly: the first revision is responsible for 8.6 of the 10.8 points. Passes four and five cost four model calls and 8.8 seconds of added latency at the median request, which is what the p50 column measures, to give back 3.1 points. This is the shape people mean by "self-critique plateaus", except that on this task and many others it does not merely flatten, it turns over.

Three of those columns behave in a way worth reading before the score column is discussed, because they are the part of the table that is not a measurement at all. The call count rises by exactly two at every step, and it has to: a pass adds one critique call and one revision call, so a run of p passes makes one generate call plus two calls for each pass after the first, which is two p minus one. The cost column then rises by exactly the same amount at every step, which is a fact rather than an arithmetic necessity. It says the two calls a pass adds cost the same at pass five as at pass two — so nothing is accumulating between passes, and the transcript is not growing under the loop. That is the chapter-1 design showing up in a price. The latency column rises by very nearly the same amount at each step too, with the first increment slightly the largest. That small drift is consistent with later critiques being shorter and equally consistent with ordinary measurement noise, and the table cannot separate the two.

Producing that table takes more discipline than running the loop does. The ticket set has to be frozen before any prompt tuning happens, or the curve measures how well the prompts were fitted to the tickets. Every pass has to be scored, rather than only the last one, which means the ledger from chapter 1 is what the harness reads rather than the loop's return value. The whole grid has to be repeated enough times to establish a noise floor, because a two-point difference between adjacent passes means nothing until you know that rerunning an identical configuration moves the mean by less than that. And the call count belongs in the table next to the score, since a quality column on its own will always argue for more passes.

The noise floor is the least obvious of those four and the one that decides how the rest is read. A score of 72.4 at pass four is not a number the system possesses; it is one draw from a distribution, and the ±0.9 figure is an estimate of how wide that distribution is. Comparing 73.2 against 72.4 without it is comparing two draws against nothing. Comparing them against ±0.9 says the difference is smaller than the instrument's own wobble, so the honest reading of passes three and four is that they are not distinguishable, while 73.2 against 70.1 is three times the floor and is. The reference you compare a difference against is the thing that makes it a finding, and the reference here has to be measured rather than assumed.

def curve(tickets: list[str], max_pass: int, repeats: int = 3):
    grid = {p: [] for p in range(1, max_pass + 1)}
    for _ in range(repeats):
        per_pass = {p: [] for p in grid}
        for t in tickets:
            run = loop(t, max_pass=max_pass)          # ledger from ch.1
            for p in run.passes:
                per_pass[p.index].append(rubric_score(t, p.draft))
        for p, scores in per_pass.items():
            grid[p].append(mean(scores))
    return {
        p: {
            "mean": mean(runs),
            "spread": max(runs) - min(runs),          # the noise floor
            "calls": 2 * p - 1,
        }
        for p, runs in grid.items()
    }

Two lines in that function carry the argument above. The inner loop reads run.passes and scores p.draft for every pass, so the grid holds a column per pass rather than one number per run — which is only possible because the loop kept its intermediate drafts as objects. And spread is the largest of the three repeat means minus the smallest, computed per pass, which is the crudest possible estimate of the noise floor and is the one worth having: it needs no distributional assumption, it is bounded below by nothing, and it is easy to explain to someone who is about to make a decision on a two-point difference.

The plateau is not the model running out of instructions#

The instrumentation from chapter 1 explains what the curve is made of. Findings do not stop being emitted at pass four — the critic produced 3.1 findings per pass at pass four against 4.2 at pass one, so it is nearly as busy — but the composition changes completely.

Check the columns before reading them, because the check is what tells you what the table is. In every column the three labelled rows add to the emitted row exactly: at pass one, 2.6 substantive plus 1.0 cosmetic plus 0.6 spurious is 4.2, and the same closure holds at passes two, three and four. That is not a coincidence; it is a statement about the labelling. The reviewer put every finding into exactly one of three buckets, so the three rows are a partition and no finding is counted twice or left out. What the closure licenses is reading any one row as a share of the emitted row, since the shares sum to one. What it does not license is treating the fifth row the same way. Re-raised findings are not a fourth bucket — a re-raised finding is also substantive, cosmetic or spurious, and is counted in whichever of the three it belongs to. At pass four the table therefore says 1.2 of the pass's 3.1 findings are repeats of pass three's, which is more than a third of them, and says nothing about which bucket those repeats fall in.

The substantive column falls by a factor of six across four passes while the cosmetic column rises, and the spurious column holds roughly flat, which means the ratio of noise to signal in the critique inverts somewhere around pass three. Locate that crossing on the table rather than taking it on trust: the two rows being compared are substantive, which is the signal, and spurious, which is the noise, and the cosmetic row is neither — it is true observations that are not worth a model call. Substantive stands at 2.6 against 0.6 spurious at pass one and 1.4 against 0.7 at pass two, so signal leads. At pass three it is 0.6 against 0.7 and the order has reversed, and at pass four 0.4 against 0.6. So a pass-three critique contains, on average, slightly more objections to correct text than real ones. A pass where 0.4 findings are real and 2.7 are not is a pass that is mostly rearranging text, and rearranging text near correct content is how the regressions in the score column get made.

The mechanism behind that composition change follows from what the critic is. The critic and the writer are one model running twice, which means both are drawing on the same weights, the numbers fixed at the end of training that decide what the model finds likely in any given context. The critique is therefore a sample from the same conditional distribution that produced the draft — the ranking over likely continuations those weights impose on whatever text is in front of them — conditioned on a prompt that asks for objections rather than for prose. Topic 4.8 established this for the single-prompt case and the loop does not change it: the errors that survive the first two passes are the ones that score well under the model's own distribution, which is exactly why they were written in the first place. A confidently wrong claim about the dispute evidence window looks, to those weights, like a well-formed claim about the dispute evidence window, whether the weights are generating it or reading it.

Each link in that chain is checkable on its own, and it is worth taking them one at a time. The writer emits a sentence because, given everything before it, those weights rank it as a likely continuation. The critic is asked whether that sentence is wrong. Answering means the weights assessing how the sentence sits against what they expect. But the sentence was chosen because it sits well against what they expect. So the critic's assessment of a claim the writer was confident about is an assessment made with the instrument that produced the confidence, and the two agree for the same reason twice. The claims that break this pattern are the ones the writer produced for some reason other than the model's expectations — a slip in a long generation, an instruction the draft did not follow, an inconsistency between two paragraphs — and those are exactly the ones the first two passes catch.

What the first two passes do catch is real and worth having, and it falls into a recognisable class: internal inconsistencies the model can detect by comparison rather than by knowledge, omissions against an explicit instruction it can check the draft against, format violations, and hedging or tone problems where the target is stated in the prompt. All of those are verifiable inside the context window, the block of text the model can see on a single call, which here is the critic prompt plus the draft plus whatever policy text was retrieved. Once they are exhausted, what remains requires information the context does not contain, and no amount of asking produces information.

There is a belief this table is built to dislodge, and it is worth stating in the form somebody would actually hold it. A team watching the critic emit findings at pass four concludes the loop is still working, because the critic is clearly still finding things. The finding count is the quantity they can see, and it barely moves — 4.2 down to 3.1 across four passes, which is the difference between busy and slightly less busy. The quantity that matters moves by a factor of six over the same span. So the visible number stays almost constant while the dangerous property varies enormously underneath it, and any monitoring built on "is the critic producing findings" will report health throughout the passes where the loop is damaging the answer.

A better critic prompt raises the ceiling and leaves the corner where it was#

The obvious response to a plateau is that the critic prompt is thin, so the next experiment is worth running properly. A second critic prompt was written with an explicit defect taxonomy of nine categories, a severity rubric with examples of each level, two worked critiques of real drafts showing the expected reasoning, and an instruction to check the answer against the retrieved policy snippets sentence by sentence. It is roughly four times longer than the original and cost 0.4 seconds more per critique call.

The richer critic is clearly better: it is 4.3 points higher at pass three, and it degrades more gently afterwards, which is itself worth something. The location of the corner did not move. Both curves get essentially all of their gain in two revisions and both stop, and the second one stops at a higher number.

Read the two tables together at the step from pass two to pass three and the arithmetic does not run the way the plateau story would predict. Under v1 the improvement between passes two and three is 71.0 to 73.2, which is 2.2 points, and under v2 it is 74.6 to 77.5, which is 2.9, so the richer critic's third pass is the more productive of the two rather than the less. That is a real qualification on the claim and it is smaller than it looks, because what the claim is about is where the curves stop rather than how they climb. By passes four and five both are flat or falling, and v2's slightly better third pass is a slightly larger share of a gain that is still exhausted in two revisions. The corner is defined by the sign change, and the sign changes in the same place for both.

That result is the answer to the most common proposal about self-critique loops, which is that the plateau is a prompt-quality problem and a sufficiently detailed critic will keep finding real issues for several more passes. A better critic prompt changes which errors the model is looking for and how carefully it looks, and it improves the yield of the passes that were already productive. It cannot give the critic a vantage point outside the weights, and the errors remaining at pass four are precisely the ones those weights do not recognise as errors. Instructing a model to look harder at a claim it believes is true produces a more thorough endorsement of the claim.

There is a second, quieter effect in the same direction. A critic prompt that names nine defect categories has been told, implicitly, that defects exist and that a good critique finds some, which raises the rate at which it manufactures them on clean text. In the taxonomy above the spurious rate under v2 was 0.9 findings per pass against 0.6 under v1 — the richer prompt found more real problems and more imaginary ones, and the net was positive here because the real ones dominated early. On a task where drafts are usually correct on the first attempt, that ratio can invert and the loop becomes a net negative from pass one.

The bound is on unaided passes#

It does not say self-critique is worthless. Ten points on the first two passes for two extra model calls is a strong return, and there are tasks — long-form drafting, code that must satisfy a spec stated in the prompt, structured extraction with a schema in context — where the first revision is close to free quality. The claim is narrower: the number of useful passes is small, bounded by something other than prompt engineering, and the curve turns over rather than flattening, so running more passes has a cost beyond the tokens.

It also does not say that all critique plateaus at two. A critique step with something in it that is not the model — a test suite result, a retrieval lookup, a compiler, an arithmetic recheck, a different model family — is a different system with a different curve, because each pass injects information the weights did not have. The single-prompt versions of those checks were covered in 4.8; the loop version is where the useful pass count actually extends, and it is what the next chapter enumerates.

Two papers are worth reading beside this curve. Reflexion: Language Agents with Verbal Reinforcement Learning is the canonical formulation of the generate-critique-revise loop with a verbal memory of past failures, and its gains come substantially from environments that return a signal — a failing test, a wrong answer flagged by the task — rather than from introspection alone. Large Language Models Cannot Self-Correct Reasoning Yet is the counterweight, showing that models asked to revise reasoning with no external signal do worse after revision than before, and that several earlier positive results had used the ground-truth label to decide when to stop revising, which is a loop with an oracle in it rather than a self-correcting one.

That second finding is worth restating as a piece of experimental design, because it is easy to reproduce by accident. If you stop the loop when the answer becomes correct, and correctness is known only from a label, then the label has entered the loop as a control signal. The system being measured is no longer the model revising itself; it is the model revising itself with a supervisor telling it when to stop, and its performance is an upper bound that nothing at request time can reach. Any harness that chooses a pass count per case rather than per configuration has the same shape and needs the same scrutiny.

The measurement in this chapter was made offline, on a frozen set, with a rubric judge and enough repetitions to know the noise floor. That is how you learn the shape of the curve, and it is not available at request time, where there are no labels, one draft per pass and a decision to make about whether to keep the revision. Everything that extends the plateau and everything that stops the loop from degrading output depends on finding a signal that does not come from the writer.