Test results become feedback only inside the loop

The patch for LEDG-2214 arrived at 09:12 and was reviewed at 16:40. It applied cleanly, every hunk at zero offset, and the file parsed — everything the previous topic's ladder asks for. The reviewer then did the thing the ladder does not do, which was to run the tests, and tests/billing/test_invoice_totals.py::test_discounted_total_rounds_once failed with AssertionError: Decimal('19.97') != Decimal('19.99'). Two cents low, which is the exact symptom the ticket had described in the first place. The reviewer wrote "this doesn't fix it, the rounding is still happening twice" in the pull request, closed the branch, and re-queued the task, and the agent started again the next morning from the original ticket text with no memory that any of this had happened.

Seven and a half hours of wall-clock elapsed between a patch that a forty-second command could have rejected and the moment anybody knew. That gap is the whole subject here. The failing assertion contained a number, an expected value, a file and a line — everything needed to make the second attempt better than the first — and all of it was spent on a human writing a sentence of prose instead. The same run, executed by the harness ninety seconds after the patch was written, would have gone back into the agent's context as text it could act on immediately.

Two words in that account carry more than they can hold alone. A test suite is the body of code a repository keeps beside its source for no purpose other than running that source on known inputs and checking what comes back: test_discounted_total_rounds_once builds two invoice line items, calls the function LEDG-2214 is about, and asserts that the answer is Decimal('19.99'). None of it ships to a customer. It exists because the only other way to know whether a change has broken something is for a person to read the change and think about it, which stops working once a codebase outgrows any one person's memory of it. You meet a suite as pytest, go test or npm test, usually run by a continuous integration server after a push and reported as a count of passes and failures.

Using that suite as the agent's verifier means the harness — the program that holds the ticket, calls the model, applies the patch that comes back and decides what happens next — runs the suite itself and reads the result as the verdict on the patch. That arrangement is what this topic is about. The tests usually already exist in the repository; what changes is who runs them, when they run, and what is done with the answer.

The habit of putting the suite at the end is inherited from how humans work, and for humans it is correct. A developer holds the change in their head for the whole afternoon, so a test result arriving at the end of the afternoon lands on a mind that still remembers what it did and why. An agent session has no such continuity: the context that produced the patch is gone once the session closes, and re-queueing the ticket the next morning starts a fresh attempt from the same prompt that produced the first one, with the same probability of producing the same mistake. Post-hoc review works against humans' strengths and against agents' weaknesses at the same time, which is why the position of the run in the pipeline matters more than the quality of the run.

A verdict at 16:40 and an instruction at 09:14#

Verifier-first design already told you that the interesting property of a rejecter is the price ratio between generating and checking. Test suites are unusually good on that measure: one patch attempt on this repository costs roughly 9,000 input-plus-output tokens and about $0.05, and the impacted-subset run that judges it — the two hundred or so tests a coverage map associates with the files the patch touched, rather than all 4,812 — costs forty seconds of CPU on a runner the team already pays for and zero tokens. What running it inside the loop adds is not cheapness, it is timing. A check that fires while the generator is still available and still holds the task in context produces a different artefact than the same check fired afterwards.

Run afterwards, a failure is a verdict. It sorts the patch into accepted or rejected, and everything else it knows is discarded when the branch is closed. Run inside the loop, the same failure is an instruction, because its text becomes the input to the next generation. The assertion Decimal('19.97') != Decimal('19.99') tells the agent that the direction of the error is downward, that the magnitude is two cents on a $19.99 total, and that the code path under test is the one the ticket named. A human writing "this doesn't fix it" throws away all three.

The last row is the honest cost of the move, and it is paid in the next two chapters rather than hidden. Nothing here makes the suite better at finding bugs; it makes the finding happen while somebody can still use it. The academic benchmark that pushed this arrangement into common practice made the same trade explicitly — SWE-bench turns a repository's own tests into the pass/fail oracle for a patch, with FAIL_TO_PASS and PASS_TO_PASS sets computed per task — the tests a correct fix has to turn green, and the tests it must leave green — and the harness runs them rather than a human reading the diff. The SWE-bench evaluation harness guide is worth reading for the mechanics of pinning an environment so the oracle is reproducible, which is a problem you inherit the moment the suite is load-bearing.

The rollback in that diagram is doing quiet work. Because patch application is already transactional at the paths the diff touched, a failed attempt costs forty seconds and one generation, and leaves a workspace identical to the one the attempt started from. Without that property the loop degrades quickly: attempt two edits a file that attempt one half-modified, the failure text stops describing the code the agent thinks it is reading, and the iteration count climbs for reasons that have nothing to do with the ticket.

The smallest complete demonstration#

Here is the whole cycle on LEDG-2214, small enough to hold in one page. The defect is in a discount helper that rounds at two different points:

# ledger/billing/pricing.py
def discounted_total(line_items, discount_pct):
    subtotal = Decimal("0.00")
    for item in line_items:
        gross = item.unit_price * item.quantity
        subtotal += gross.quantize(Decimal("0.01"), ROUND_HALF_UP)
    discount = (subtotal * discount_pct / 100).quantize(
        Decimal("0.01"), ROUND_HALF_UP
    )
    return (subtotal - discount).quantize(Decimal("0.01"), ROUND_HALF_UP)

The test that catches it has been in the repository since before the agent existed:

def test_discounted_total_rounds_once():
    items = [LineItem(Decimal("6.664"), 2), LineItem(Decimal("6.665"), 1)]
    assert discounted_total(items, Decimal("0")) == Decimal("19.99")

The two line items are $6.664 and $6.665, which sum to $19.993 and quantize to $19.99. Rounding each item first gives $6.66 plus $6.67, and the two cents the ticket is about are lost before the total is ever computed. The harness runs the subset, gets one failure, and sends the agent a record built from it:

FAILED tests/billing/test_invoice_totals.py::test_discounted_total_rounds_once
  assert discounted_total(items, Decimal("0")) == Decimal("19.99")
  E  AssertionError: Decimal('19.97') == Decimal('19.99')
  ledger/billing/pricing.py:14  in discounted_total
  tests/billing/test_invoice_totals.py:31  in test_discounted_total_rounds_once

The first revision the agent produced removed the quantize inside the loop, which is correct, and also removed the one on the final return, which is not. The function then returned Decimal('19.9930'), test_discounted_total_rounds_once still failed — now with Decimal('19.9930') == Decimal('19.99') — and two tests in tests/billing/test_invoice_render.py that format the total into a PDF line broke as well, because the renderer assumed two decimal places and produced $19.9930. Three failures where there had been one is a worse outcome by test count and a better one by information content, because the record now names a file the agent had not been thinking about and pins a requirement nobody wrote down anywhere: the public return value is quantized exactly once, at the end. The third attempt kept the interior rounding removed and restored the final quantize, and the subset came back with 214 passed in 41 seconds.

Something in that sequence is worth noticing, because it is the mechanism the rest of the topic depends on. The agent did not reason its way to the renderer's assumption; it was told about it by a test it broke. In-loop execution converts the repository's accumulated, undocumented invariants into text at the exact moment the agent needs them, which is the closest thing the pattern has to a superpower — and the reason a repository with a weak suite gets much less out of this arrangement than the loop's structure suggests.

The loop that produced this is not complicated, and the parts worth looking at are the ones that decide what gets fed back:

def solve(ticket, workspace, max_attempts=4):
    feedback = None
    for attempt in range(1, max_attempts + 1):
        patch = agent.generate(ticket, workspace, feedback=feedback)
        snapshot = workspace.apply(patch)          # transactional, per 13.5
        result = runner.run(workspace, suite="impacted")
        if result.passed:
            return Accepted(patch, attempts=attempt)
        workspace.rollback(snapshot)
        feedback = build_failure_record(result)    # chapter 2 lives here
    return Exhausted(ticket, attempts=max_attempts, last=feedback)

The attempt cap matters more than it looks. Four is not a magic number; it is where this team found the marginal attempt stopped paying, since across a sample of 300 tasks the share solved on attempt four was 3%, and attempts five and six between them added under 1% while contributing most of the runs where the agent started rewriting things it had already got right. Exhaustion is a real outcome and the loop should say so plainly: Exhausted carries the last failure record forward so the human who picks the ticket up begins where the agent stalled rather than at the ticket text. A loop that silently returns its final attempt as if it had succeeded is worse than no loop, because it launders four failures into one plausible-looking patch.

Three iterations at 9,000 tokens each is about $0.15 of model spend and roughly two minutes of runner time for a fix a human would have reviewed in ten minutes and re-queued in seven and a half hours. The economics are not the argument on their own, since a team could simply review faster. The argument is that the third attempt was only possible because the first two failures were still addressable when they happened, and that property does not survive being moved to the end.

What a green subset is not#

A loop that ends green has established exactly one thing: the patch satisfies the assertions that ran. It is worth being blunt about the claims that do not follow, because every one of them gets made in practice.

It is not a claim that the ticket was implemented. Nothing in test_discounted_total_rounds_once knows what LEDG-2214 asked for; it happens to encode the same requirement because someone wrote it that way years earlier. A patch that made that test pass by special-casing the two-item input would have produced the same green.

It is not a claim of correctness on untested paths, and on this repository that gap is large — line coverage, the share of executable lines that at least one test runs, is 41% and concentrated in the utility packages, so a service-layer change frequently runs no assertions that touch it at all. A green run on a change to ledger/services/settlement.py may mean the tests passed around the change rather than on it.

It is not continuous integration, and treating it as a replacement is the substitution that causes real damage later. The in-loop run is fast and narrow by design, executing on a scratch workspace against whatever the base commit was when the task started. Something exhaustive still has to run against the merge result, and the reasons that stays true are specific enough to be worth their own chapter.

Most importantly for everything that follows, it is not independent of the agent unless somebody arranges for it to be. In this chapter the failing test was written by a human, lives in a file the agent was told not to modify, and runs under a command the harness controls. Change any one of those and the green stops being evidence. That is not hypothetical: an agent that can edit test_invoice_totals.py has a much cheaper route to green than fixing quantize, and an agent that chooses which tests run has a cheaper one still.

What the loop still has to be told#

The shape is now fixed — generate, apply, run, feed the failure back, roll back, repeat — and every remaining chapter of this topic constrains one of its arguments rather than changing its structure. The first argument to constrain is the one the code above waves at with a function name: build_failure_record(result). On this run the result was one assertion in one file, and piping the runner's raw stdout back would have worked fine. On a failing run of the full suite the same call returns something closer to 30,000 tokens of tracebacks, warnings and progress dots, and what happens to the agent's next patch when you send it all of that is measurable, expensive, and not what most teams expect.