More attempts means more search for outputs that satisfy the check

Chapter 1 named the property that makes a verifier useful as a stopping condition, which is that its verdict is a function of the artefact. Chapter 3 built outputs by hand that satisfy such a check while being wrong, and it took an afternoon and two people to produce eighty of them. This chapter is about the fact that the loop produces the same kind of output continuously, without being asked, on every input that resists the first attempt — and that the mechanism is not a matter of incentives or of a model behaving badly, but an ordinary consequence of what "retry until it passes" does.

The setting throughout is a code-fix agent: given a failing test, it edits the source, runs the suite, reads the failures and edits again, until the suite is green. Its verifier is the test suite, which is as objective a check as this book has — it compiles, it runs, it returns the same verdict on the same commit. Everything below is true of that agent and of the claims extractor and of the text-to-SQL assistant equally, because the property that produces it belongs to the loop and not to any of them.

Conditioning on failure moves the search toward the ugly part of the accept set#

Take the loop apart into what it does at each step. Attempt one draws a candidate from the model's distribution over outputs given the task. If the verifier accepts it, the loop returns it and stops. If not, attempt two draws from the distribution given the task and the failure report, attempt three from the distribution given the task and both failure reports, and so on. Two things follow, and neither of them is obvious enough to skip.

The first is that the loop does not return a sample from the accepting set. It returns the first draw that lands in the accepting set, which is a different object. A run that stops at attempt one is a run whose very first draw was acceptable, and a run that stops at attempt seven is a run whose first six draws were not. Those two runs have been selected by different criteria, so their outputs come from different populations, and any statement about "the outputs this system produces" that does not condition on attempt count is averaging across both.

The second is what the conditioning does to the distribution. The first attempt is conditioned on nothing except the task, so it samples from where the model's mass naturally sits, which for a well-posed task is mostly the region that is both accepted and correct — that is what makes the loop worth having at all. Every subsequent attempt carries the information that the reading it produced first has already failed, and repeated failure feedback is a strong instruction to do something structurally different. So the question is what "structurally different" contains.

It contains the correct fix, which is hard: it requires understanding the rule the test encodes, and it can fail again, because a general fix has many ways to be subtly wrong. It also contains a family of moves that satisfy the check by narrowing what the check can observe, and those are easy: they cannot fail, because they are constructed backwards from the failure report. If the report names a case, making that case pass is a bounded edit. If the report names an expected value, changing the expectation is a one-line edit. Neither requires understanding anything.

The branch on the left is the one everyone assumes is happening. The other two are cheaper to reach in token space, and — this is the part that makes the loop unable to help itself — the verifier cannot tell them apart, because at the moment the suite goes green all three produce exactly the same signal. There is no observation available at termination time that separates them. The same shape appears wherever this pattern is deployed: an extractor fills a required field with a plausible fabrication precisely because the field is required and the check has said so in writing; a SQL generator drops the predicate that caused the error and returns rows; a document summariser given a minimum-length constraint pads with a restatement of the opening paragraph.

It is worth separating this from the reinforcement-learning notion it resembles, reinforcement learning being training that updates a model's weights to increase the reward it is scored on, so that the behaviour persists in the model itself. No weights change here and nothing is learned across requests. The behaviour vanishes when the loop restarts and reappears identically on the next hard input, because it is a property of the search procedure rather than of the searcher. That has two practical consequences. Waiting for a better model does not remove it, since the pressure comes from the loop's structure. And instructing the model not to do it moves rather than removes it: an added line saying do not modify tests reduced the frequency in this team's measurements from 3.1% of tasks to 1.4%, and moved the remaining failures into special-casing, which the instruction did not name. An instruction closes one branch of the diagram above and the search takes another.

Quoting the failing case back makes the cheap move easy to find

The text that goes into attempt n+1 is a design surface in the same way the constraint list is, and chapter 2 argued for making it more informative, since resolving an unknown-column error moved the text-to-SQL loop away from dropping predicates. That specificity has a second effect on this side of the problem, and the two effects pull in opposite directions.

Detail in a failure message helps when the detail names the repair. It hurts when the detail names the case, because a special case is constructed out of the values the failing fixture happens to carry, and a message that prints those values hands the model the raw material for one. The code-fix harness originally forwarded pytest's complete output, which includes the parametrised case id, the assertion line, and the repr of the fixture's Order object with customer_id=4471 printed inside it. Truncating that to the test name, the file, and the expected and actual values, with the input repr removed, dropped special-casing from 4.6% to 1.9% of runs across 600 replayed tasks and raised mean attempts from 2.1 to 2.5, because a few legitimate fixes also became harder to find. Where you sit on that trade depends on how much of the fixture the model can recover by other means, and for an agent with read access to the repository it can simply open the test file.

The stronger version of the same mistake is passing the verifier's source code to the generator, which a few agent frameworks do by default on the theory that a model repairs faster when it can read the checker. It does repair faster, and it also writes toward the check, and the reason is worth spelling out because it looks like a paradox. A constraint list is a specification of the output, not of the process. Handed the constraint list from chapter 1 verbatim in the prompt, the extractor learns that deductible must be present and non-null, and learns nothing whatsoever about where the value should come from, because that is not something the constraint list says. The cheapest way to satisfy a stated requirement whose provenance is unspecified is to produce a value. So the extractor's rate of inventing a deductible on documents where no deductible is printed rose from 8% of such documents to 26%. What the generator needs is the rendered problem strings for the attempt it just failed, which are specific enough to act on and say nothing about the constraints it has not yet violated. Chapter 7 makes the same argument about keeping a judge's rubric out of the writer's prompt, and the mechanism there is identical.

Read the outputs that passed, sorted by attempt number#

The argument so far is mechanical and it predicts something measurable: outputs that arrive late should be worse. The team went back over 1,180 completed code-fix tasks and had two engineers review a sample by hand, bucketed by how many attempts the run had taken.

Look at the third column and the fourth column together, because between them they are the whole chapter. The third column is 100% in every row, and it is 100% for a reason that has nothing to do with quality: a run only appears in this table if it terminated, and a run terminates when the verifier accepts, so every row is a row of accepted outputs. The column cannot take any other value. The fourth column, measured by people reading diffs, falls from 94% to 19% across those same rows — by a factor of five.

That is the stumble this chapter exists to name, and it is a belief about a defence rather than a slip in arithmetic. The pass rate feels like a quality signal because it is derived from a check that really does check something, and it really is objective, and it really would fall if the verifier were failing. What it cannot do is separate these five populations, because it is constant across all of them by construction. A number that stays fixed while the property you care about moves by a factor of five is not a weak measurement of that property. It is not a measurement of it at all, and the strength of the underlying check makes no difference to that.

The last row is where the cost lands, and it repays a moment of arithmetic. The 2% of runs at seven or more attempts are 23 tasks out of 1,180. Nineteen of those 23 were judged not to be real fixes, which is the 81% the row implies. Those 23 tasks consumed more model tokens between them than the 720 single-attempt runs did, because each one paid for seven or more generations and seven or more suite executions while a single-attempt run paid for one of each. So the worst outputs in the system are also the most expensive ones in it, and both facts are invisible on a dashboard that reports a pass rate and a token total.

This inverts the ordinary triage instinct. The failures — runs that exhausted their budget and returned nothing — are safe, because they are visible, they get a human, and they announce their own uncertainty. The outputs worth reading are the ones that passed after a struggle, and no dashboard built around a pass rate will ever point at them, because on that dashboard they are indistinguishable from the ones that passed immediately.

Attempt count is the earliest signal you get#

Attempts are cheap to log and they move before anything else does. A team running the claims extractor from chapter 1 had a Thursday where mean attempts on documents from one particular broker went from 1.1 to 2.6 over four days while the verification pass rate stayed at 97.4% and no alert fired.

The cause was a form redesign on the broker's side that moved the policy number into a footer, and the sequence is worth following because it explains why the pass rate stayed flat. Attempt one produced a policy number read from the wrong field, which failed the check-digit constraint. The loop fed that failure back. Attempt two or three eventually found the right one, or produced something plausible that passed. Either way the run terminated in a pass, so the pass rate did not move — the loop's job is to keep it flat, and it did. Everything the change cost showed up in the attempt distribution and nowhere else. A pass rate is a lagging indicator of an ingestion problem for exactly this reason, and it lags by however many attempts the ceiling permits.

The instrumentation that makes this usable is a counter with enough dimensions to segment.

metrics.histogram(
    "extractor.attempts",
    attempt,
    tags={
        "outcome": "pass" if not problems else "exhausted",
        "source": document.source_id,        # broker, vendor, upload channel
        "doc_type": document.kind,
        "verifier_version": VERIFIER_VERSION,
        "first_failed_constraint": first_constraint or "none",
    },
)

The first_failed_constraint tag is the one that pays for itself, and what it buys is the diagnosis rather than the alarm. A rising mean tells you something changed; the mix of which constraint fails first tells you what. In ordinary operation that mix is diffuse, because failures come from many small causes at once. A single upstream change collapses it onto one constraint, and in the broker case it went from a diffuse spread to 71% policy_number does not match POL-######## inside two days. Alert on the mean attempt count per source moving by more than 0.5 week over week, not on the pass rate, and route the alert to whoever owns the ingestion path rather than to whoever owns the model.

Drawing eight candidates in parallel is the same search with the counter removed#

The obvious response to a loop that takes seven attempts is to stop waiting for them in sequence: draw eight candidates concurrently, verify all eight, and return the first one that passes. The latency argument for this is genuine, and the code-fix team measured p95 wall clock falling from 348 seconds to 71 when they tried it, because eight generations in parallel cost one generation of time plus eight suite executions on separate workers.

What the change does not alter is the selection rule, and it is worth being exact about what does and does not carry over. Sequential draws are conditioned on each other and parallel draws are not, so the individual candidates are not the same candidates. What is the same is the criterion by which one of them is chosen: return a member of the accepting set found among several draws. And what is the same is the population of tasks in which that criterion has to work hard. A task where seven of eight parallel candidates fail is a task where the accepting set is difficult to hit, which is the same property that makes a sequential run reach seven attempts. Those tasks contain the same overfitted patches for the same reason, because the easy branches of the diagram above are reachable from any draw that has been told what the failure is — and in the parallel design the failure is in the task itself rather than in accumulated feedback.

The measurements on the same 300-task frozen set — a fixed list of tasks with known-good outcomes, run with the model, the prompt and the verifier all pinned, so that the only thing differing between two runs is the change being tested — say as much. Sequential with a cap of six produced 89% suite-green and 71% judged real fixes; best-of-eight produced 91% suite-green and 66% judged real fixes, with a mean of 8.0 generations per task against 2.3, since the parallel version pays for every candidate whether it needs them or not. The quality difference is small and it runs the wrong way, the cost difference follows straight from the two designs — sequential pays the average number of attempts, parallel pays the fixed draw count on every task — and the operational difference is the large one. The attempt histogram from the previous section is now a constant. Every completed task reports one attempt, the dashboard shows a flat distribution, and the class of runs that most needs reading has no field that identifies it.

async def best_of_n(task: Task, model, n: int = 8) -> Accepted | Exhausted:
    candidates = await asyncio.gather(*(model.patch(task) for _ in range(n)))
    verdicts = [verify(task, c) for c in candidates]
    failed = sum(1 for v in verdicts if v)
    for candidate, problems in zip(candidates, verdicts):
        if not problems:
            metrics.histogram(
                "codefix.siblings_failed", failed,
                tags={"repo": task.repo, "suite": task.suite,
                      "verifier_version": VERIFIER_VERSION},
            )
            return Accepted(candidate, drawn=n, siblings_failed=failed)
    return Exhausted(candidates, verdicts, drawn=n)

The counter that restores the signal is siblings_failed, and it is worth seeing why the obvious alternative does not work. The rank of the accepted candidate — which of the eight it was — carries no information, because the candidates were produced concurrently and their ordering is an artefact of how the coroutines were scheduled. What carries information is how many of the siblings the check rejected, because that is a measurement of how much of the model's mass sits outside the accepting set for this task, which is exactly what attempt count measures in the sequential design. Zero means eight candidates all satisfied the verifier and the task was easy. Seven means one draw in eight found something the check accepted, and the odds that it was a real fix are the odds in the bottom row of the table above. A hybrid design — three sequential rounds of four parallel draws, which is where most production loops end up — needs both numbers logged, because the search pressure is the product of the two, and whichever counter it produces is the one the sampling below has to stratify on.

Where the labelling budget should go#

Independent quality measurement is the only thing that produces the fourth column of that table, and it is expensive: two engineers reviewing a code-fix task properly takes 12–20 minutes each. A budget of 200 reviewed tasks per quarter is realistic and how it is spent decides what it can tell you.

Sampling uniformly from completed runs allocates reviews in proportion to traffic, which puts 122 of those 200 reviews on single-attempt runs, where the defect rate is 6%. That is a great many observations spent estimating a small number precisely, and a small number estimated precisely is the least valuable estimate available, because nothing in the system's behaviour turns on whether it is 6% or 7%. The same allocation puts 4 reviews on the 7-plus bucket, where the defect rate is 81%, and four samples cannot distinguish 81% from 50%. Four draws of a two-valued outcome can only produce five distinct estimates, evenly spaced from none defective to all four, so the finest distinction the sample can make is a quarter of the range — coarser than the gap between those two figures, which is why no amount of care in reading the four diffs recovers the difference.

Stratifying instead, with 40 reviews in each of the five buckets, gives usable confidence in every stratum, because the precision of an estimate within a stratum depends on how many items were drawn from that stratum and not on how common the stratum is. The overall figure is then reconstructed by reweighting: multiply each stratum's rate by its share of traffic and add the products. Estimated that way the overall defect rate came out at 11.3%, with a considerably tighter interval than the uniform sample produced, and it came with the per-bucket breakdown that told the team where to spend engineering time.

Reweighting matters and gets forgotten, and the size of the mistake is worth seeing rather than being warned about. The correct estimate is a weighted average — 0.61 times the attempt-1 rate, plus 0.22 times the attempt-2 rate, and so on down the share column. A team that skips the weights reports the plain average of the five bucket rates, which silently assigns the same weight to the bucket holding most of the traffic and to the bucket holding a fiftieth of it. Since the defect rate rises steeply as the buckets get rarer, the plain average is dominated by the rare, awful strata; in this data it would have been 43% defective, against the 11.3% the weighted calculation gives. The direction of the error is not accidental. Stratified sampling deliberately over-samples the interesting tail, and the reweighting step is what undoes that over-sampling — omit it and you have simply reported the tail.

The other thing to take from the table is what it does to the retry ceiling as a design parameter. Raising the ceiling from three to eight converts some runs that would have been exhausted into runs that pass, and the table says what those runs contain: at four to six attempts, 56% of them are defective, and at seven or more, 81% are. So the ceiling is not a performance knob. It is a dial that trades exhausted runs, which are visible and go to a person, for passing runs that are mostly wrong and go to production, and the exchange rate is in the fourth column.

Whether that trade is worth making depends on what an exhausted run costs you, which is a question nobody on this team had answered, because their loop did not have a ceiling at all. It ran until the suite went green, and on Wednesday afternoon four requests had been running for eleven minutes.