A retry loop is only as honest as its diagnostics

The task was routine: a .proto file — the schema for a Protocol Buffers message, from which a generator writes the classes every service uses to read and write it — gained a discount_cents field, and somebody had to regenerate those classes, the stubs, and update the four call sites in the Kotlin order service. The agent had a twelve-attempt budget and a Gradle daemon for feedback, a build process that stays alive between runs and redoes only the work whose declared inputs have changed, which returned in about 40 seconds per attempt instead of the 3 minutes a clean build costs. It used all twelve, took 26 minutes, and produced this on every single one:

e: OrderRouter.kt:88:34 unresolved reference: discountCents
e: OrderRouter.kt:91:12 type mismatch: inferred type is Int? but Int was expected
e: PricingClient.kt:44:19 unresolved reference: getDiscountCents

Character for character the same three errors, twelve times. The diffs did not stay routine. Attempt three added a local extension property val OrderProto.discountCents: Int get() = 0. Attempt six wrapped the call site in a null-coalescing chain that could not be null. Attempt nine introduced a LegacyOrderShim class with hand-written accessors mirroring fields the generated code already had. Attempt twelve deleted the pricing call entirely and left a TODO. Then an engineer ran ./gradlew clean build on the attempt-one tree and got BUILD SUCCESSFUL in 3m 12s.

The stubs had been regenerated correctly on attempt one. The Gradle task that compiles Kotlin declared its inputs as the src/main/kotlin tree, and the protoc plugin wrote into build/generated/source/proto, which that task's input fingerprint did not cover in this project's configuration. The daemon therefore concluded that nothing relevant had changed, skipped the work, and replayed the diagnostics it had cached from before the regeneration. Every error the agent read described a tree that had stopped existing 26 minutes earlier, and every patch it wrote was a correct, competent repair of a problem that was already gone.

The assumption the loop was making#

The loop's logic was ordinary and its assumption was unstated: that the diagnostic stream is ground truth about the current working tree. Everything downstream depends on that. Retry only makes sense if a failure describes the artifact you are about to modify, and the whole reason code is an attractive agent domain — a cheap verdict on this output, right now — evaporates when the verdict is about a different output.

What makes this hard to spot from inside the loop is that the agent's behaviour was locally rational at every step. It read a diagnostic naming a file and a line, made an edit that would plausibly resolve it, and observed that the error persisted, which is normally evidence that the edit was insufficient rather than evidence that the observation was fictional. Escalating from a small edit to a larger one is the correct response to that evidence. The system had no way to represent the hypothesis that the feedback itself was wrong, so it kept spending attempts on the only hypothesis it could represent.

Three ways feedback lies#

Stale feedback is the most spectacular of the three failure modes, and the other two are more common.

Incomplete feedback comes from tools that stop early. pytest -x reports the first failure and exits, so an agent that broke six tests sees one, fixes it, sees the next, and reads a sequence of six single-failure runs as steady progress toward a goal it may be moving away from. Kotlin's compiler reports a bounded number of errors per file; mypy with --follow-imports=skip silently declines to check the module where the real contract lives. None of these tools are wrong, and each of them turns the agent's picture of the tree into a keyhole view.

Misattributed feedback names the site where the symptom surfaced rather than the site of the cause. A missing package in the virtualenv produces ModuleNotFoundError: No module named 'orjson' at an import line the agent will happily start rewriting; the fix is pip install, and nothing in the message says so. A leftover .pyc, a stale node_modules, a database left in a bad state by attempt four — each of these produces a diagnostic that is technically accurate and points the agent at the wrong file.

The loop, with the safeguards in it#

The minimal honest loop is short enough to read in full. It generates a patch, applies it in an isolated workspace, runs the checks, and either accepts, retries from the diagnostics, or stops — with the three safeguards above expressed as code rather than as intentions.

import hashlib, subprocess
from dataclasses import dataclass

@dataclass
class CheckResult:
    ok: bool
    output: str
    files: frozenset[str]

    @property
    def fingerprint(self) -> str:
        return hashlib.sha256(self.output.encode()).hexdigest()[:12]

def run_checks(workdir: str, *, no_cache: bool = False) -> CheckResult:
    cmd = ["./gradlew", "--offline", "build"]
    if no_cache:
        cmd = ["./gradlew", "--no-daemon", "--rerun-tasks", "clean", "build"]
    proc = subprocess.run(cmd, cwd=workdir, capture_output=True, text=True, timeout=900)
    out = proc.stdout + proc.stderr
    return CheckResult(proc.returncode == 0, out, parse_error_files(out))

def attempt_task(task, workdir: str, *, max_attempts: int = 8) -> str:
    history: list[CheckResult] = []
    for n in range(1, max_attempts + 1):
        patch = generate_patch(task, history)
        apply_patch(workdir, patch)               # workdir is a fresh git worktree
        result = run_checks(workdir)

        if result.ok:
            confirmed = run_checks(workdir, no_cache=True)
            return "accepted" if confirmed.ok else escalate("green only with cache", n)

        if history and result.fingerprint == history[-1].fingerprint:
            fresh = run_checks(workdir, no_cache=True)      # is the feedback real?
            if fresh.ok:
                return escalate("stale diagnostics: clean build passes", n)
            result = fresh

        if not (result.files & patch.touched_files) and n > 1:
            return escalate("diagnostics disjoint from patch", n)

        history.append(result)
    return escalate("retry budget exhausted", max_attempts)

Three things in that listing carry most of the weight. The workspace is a fresh git worktree per task — a second checkout of the same repository, sharing its history and carrying its own files and build directory — which means attempt four cannot poison attempt five through a leftover build directory, and two tasks running in parallel cannot interleave their edits — running agents in a shared checkout produces exactly the misattributed diagnostics described above, and it is the single most common way teams manufacture this problem for themselves. The retry budget is a small integer rather than "until it passes", because an unbounded loop on a lying signal is how you spend $40 of tokens producing a LegacyOrderShim. And a repeated diagnostic fingerprint triggers a cache-free re-run rather than another patch, which is the specific safeguard that would have ended the protobuf episode at attempt two instead of twelve, at a cost of one 3-minute clean build.

The green path carries a confirmation run for the same reason, and it is the more valuable of the two. A false red wastes attempts; a false green ships a patch that never actually compiled, and the fastest way to get one is a build cache that returns success for a target whose inputs it is misjudging in the other direction.

Where the budget should stop, and what goes back into the prompt#

The retry limit is usually picked by feel, and it is one of the few parameters in the loop you can set from data after a week of running. Log the attempt number on which each task was finally accepted, and the distribution is steep. On 340 tasks through the Kotlin service's loop, with an 8-attempt budget, it came out like this:

Beyond attempt four the loop rescues seven tasks out of 340 and spends four attempts on each of the 31 tasks it was never going to finish, which is about 80 minutes of compute to convert seven failures into successes and 31 failures into slower failures. A budget of four is defensible on those numbers and a budget of twelve is not, and the deeper reason to keep it small is that the tail is where the strange diffs live — the attempt-nine shim class is not a random event, it is what generation looks like when the plausible repairs are exhausted and the model is still being asked for one.

What you feed back matters as much as how many times. The tempting implementation passes the whole build log into the next prompt, which for Gradle is 4,000 lines of task graph and dependency resolution wrapping the three lines that matter, and which costs somewhere near 12,000 tokens per attempt for a signal that fits in 60. Parsing the tool's output into structured diagnostics — file, line, severity, message — and passing those, plus the diff that produced them, gives the generator a cleaner input and gives you the fingerprint the stall detector needs. It also makes a second thing possible: passing the previous attempts' diagnostics alongside the current ones, so the model can see that its last three edits changed nothing, which is information it otherwise has to infer from a conversation it may have had truncated out from under it.

What this is not an argument against#

None of this says to run clean builds. The economics that created the daemon are real — 40 seconds against 3 minutes 12 seconds is the difference between eight attempts costing 5 minutes and eight attempts costing 26, and the entire case for putting a check inside the loop rests on it being cheap. The position that works is to treat incremental results as a fast, usually-correct oracle and to require confirmation from a hermetic run — one that starts from a clean tree with every cache and daemon disabled, so its verdict depends on nothing but the files on disk — at exactly two moments: before declaring success, and before declaring failure. That is two slow builds per task rather than eight, and it recovers the honesty without giving up the speed.

Nor is this a story about the model doing something strange. The escalating patches were a reasonable response to the evidence presented; a competent engineer given the same three errors, refreshed after each edit, would eventually try stranger things too. Treating this as a hallucination problem sends teams to prompt engineering, when the defect is in the build configuration and in a loop that had no way to doubt its own input. Daniel Demmel's Feedback loop engineering makes the general form of the point: agent quality tracks the speed and fidelity of the checks, and fidelity is the half that quietly fails, because a fast wrong signal looks exactly like a fast right one from inside the loop.

There is a residual case the safeguards do not cover, and it is worth naming rather than discovering. Flaky tests produce diagnostics that are honest about the tree and non-deterministic anyway, so a fingerprint comparison sees two different failures and reads them as progress. The usable defence is to quarantine known-flaky tests out of the loop's check set entirely and let them run in CI where a human interprets them, which costs you the failure classes those tests cover and buys back a signal the loop can reason about.

With the feedback honest, the remaining question is which checks belong inside the loop at all. The confirmation runs in the listing above hint at the tension: an 8-second check and an 11-minute one cannot both sit on every attempt, and choosing between them is not a matter of taste.