Warnings a patch did not cause are noise the agent will still obey

Lint went into the loop on a Tuesday, and by Friday the median task time had gone from 6.5 minutes to 13.4. The tasks themselves had not changed. What changed is visible in the trace of a ticket that asked for a currency field on the invoice response: the agent's first four tool calls after reading the file were edits to packages/api/src/routes/reports.ts, packages/api/src/middleware/auth.ts and two files under packages/worker, converting let to const, deleting an unused import, and adding a return type to an exported function. None of those files are in the ticket. None of them were touched by the patch. All four edits were correct, in the sense that the linter had genuinely complained about each of them, and every one of the complaints predated the agent by about eighteen months.

The repository carries 312 lint findings on a clean checkout. The loop feeds the linter's full output back on every attempt, so on every attempt the agent receives 312 well-formed, precisely located, apparently actionable problems, of which zero were caused by the work in progress. Instruction-tuned models — models given a round of training specifically on following instructions in their input, rather than only on continuing text — are built to be responsive to what is in front of them, and a list of problems in the context is functionally a list of tasks. The agent is not malfunctioning when it starts fixing them; it is doing the most obvious thing available given what it was handed.

Precision is relative to the question you asked#

Applied to the repository, that linter has excellent precision, meaning the share of what it reports that is genuinely a finding rather than a mistake: 312 findings, essentially all of them true statements about the code. Applied to the question the loop is actually asking, which is what did this patch break, its precision is approximately zero and its false-positive rate — the share of what it reports that is not an answer to that question — is approximately total. Both descriptions are correct at once because they score against different ground truths, and the loop only cares about the second one. A signal's value in an agent loop is never a property of the tool in isolation; it is a property of the tool plus the question, and the question inside a retry loop is always about the change under construction.

That framing gives four axes worth tracking for any candidate check, and they come apart from each other more than people expect. There is diagnostic precision, meaning how exactly a finding names the thing to edit — a path, a line and a column at one end, "the integration suite failed" at the other. There is false-positive rate relative to the task, which is what this chapter is about. There is runtime, which chapter 6 prices properly. And there is actionability, meaning whether acting on the finding is a patch the agent can write at all, as opposed to context that merely informs it. A check can score well on three and be useless on the fourth: the linter above is superbly precise, fast, and actionable, and its task-relative false-positive rate ruins it.

The baseline is a set difference#

The fix is to ask the question the loop means. Run the check against the tree as it was before the edit, run it again after, and return the difference. Everything in the second run that was not in the first is what this patch introduced; everything present in both is the repository's standing debt, which is a real problem and somebody else's ticket.

The naive implementation keys findings on (path, line, rule) and breaks immediately, because line numbers move. Inserting four lines near the top of a 200-line file with twelve standing warnings below the insertion point produces a diff claiming twelve findings were fixed and twelve new ones introduced, and hands the agent all twelve as its own doing. The repair is to key on content rather than position:

import hashlib, re, collections

NUMBERS = re.compile(r"\b\d+\b")
WS = re.compile(r"\s+")

def fingerprint(diag: dict, source: list[str]) -> str:
    """Position-independent identity for a diagnostic."""
    idx = diag["line"] - 1
    code_line = source[idx].strip() if 0 <= idx < len(source) else ""
    parts = [
        diag["tool"],
        diag["code"],                                  # rule id / error code
        diag["path"],                                  # repo-relative
        NUMBERS.sub("N", WS.sub(" ", diag["message"])),
        WS.sub(" ", code_line),                        # the offending source
    ]
    return hashlib.sha1("|".join(parts).encode()).hexdigest()[:16]

def keyed(diags: list[dict], read_source) -> dict[str, dict]:
    """Fingerprint, disambiguating repeats within a file by occurrence."""
    seen, out = collections.Counter(), {}
    for d in diags:
        fp = fingerprint(d, read_source(d["path"]))
        seen[fp] += 1
        out[f"{fp}#{seen[fp]}"] = d
    return out

def attribute(before: list[dict], after: list[dict], read_before, read_after):
    b, a = keyed(before, read_before), keyed(after, read_after)
    return {
        "introduced": [a[k] for k in a.keys() - b.keys()],
        "resolved":   [b[k] for k in b.keys() - a.keys()],
        "standing":   len(a.keys() & b.keys()),
    }

Normalising numbers out of the message matters more than it looks: a message like Line 214 exceeds maximum length of 100 embeds a position that will shift, and without the substitution every such finding is reported as new after any edit above it. Including the source line in the key is what makes the fingerprint survive movement while still distinguishing two different prefer-const findings in the same file. The #n occurrence suffix handles the case where a file has three identical findings on three identical lines, so that going from three to four is reported as one introduced finding rather than none.

The resolved list is worth keeping even though nobody asked for it. It tells you when a patch cleared standing debt, which is the input to a ratchet policy — the baseline count may go down and may never go up — and it catches the case where the agent "fixed" a warning by deleting the code that produced it.

The cost is one extra run of the check, and it is cheaper than that implies, because the pre-edit run happens once per task rather than once per attempt. On a task that takes 3.2 attempts, a 2-second linter costs 6.4 seconds of in-loop time plus 2 seconds of setup, and the setup run is also the thing that tells you whether the repository was clean before the agent started.

The two rows people reach for first are the cheap ones in the middle, and they fail on the case that matters most. Changing a function's signature introduces a no-unsafe-argument finding at every call site, none of which are in the changed file and none of which are on changed lines. A strategy that filters by location will silently drop exactly the findings that describe the damage the patch did, which is a worse failure than showing too much, because it is invisible.

The checked-in baseline file — the approach taken by ratchet tools that store a snapshot of accepted findings in the repository — avoids the extra run and is the right choice when the check is slow enough that running it twice per task hurts. Its cost is that the file drifts: it is updated by whoever last remembered to, and a finding that was legitimately fixed and then reintroduced looks like part of the baseline. For checks under ten seconds, computing the baseline fresh at the start of each task is simpler and always correct.

What the switch buys is measurable on the same tasks that motivated it. On the week after attribution went in, the linter's contribution to the feedback payload fell from 312 findings to a median of zero and a 90th percentile of two, median task time came back to 6.9 minutes against the 6.5 from before lint existed, and the share of accepted patches containing an edit to a file outside the task's blast radius fell from 41% to 3%. The residual 3% is mostly legitimate — a shared helper the change genuinely needed — and the residual 0.4 minutes is the baseline run. That is the shape of the win: the check keeps everything it was contributing and stops charging the loop for the repository's history.

What the harness knows and the model cannot#

The reason this belongs in the pipeline rather than the prompt is the same structural reason as dependency ordering, arriving through a different door. Whether a warning predates the patch is a fact about a run of a tool against a tree that no longer exists. The model was not present for that run, cannot see that tree, and has nothing in its context from which the answer could be derived. An instruction telling it to ignore pre-existing warnings asks it to perform a comparison against data it does not have; what it will actually do is guess from plausibility — this file looks unrelated to my task, so probably not mine — which is a heuristic that fails precisely on the signature-change case above.

The harness, on the other hand, has both runs and needs one set difference. When a fact is available to the harness and structurally unavailable to the model, putting the decision in the prompt is a choice to compute it badly.

Baselining is not cleanup, and it can hide things#

Attribution makes the standing findings invisible to the loop and leaves all 312 of them in the repository. That is the correct division of labour for the inner loop, and it is not a solution to lint debt, which needs its own campaign with its own gate — the ratchet on the baseline count, so that debt can shrink and cannot grow. There is production experience with running that campaign using agents: BitsAI-Fix reports a pipeline that takes lint findings, generates and validates repairs, and lands them at scale as their own change, which is the right shape because it puts the debt work in a loop whose task is the debt, where the findings are on-task by construction.

Attribution can also hide a genuine regression, in a way worth knowing about. If a patch introduces a finding whose fingerprint collides with one already in the baseline — the same rule, the same file, an identical source line — the occurrence counter catches it only if the count changes, and there are configurations where a fix and a break cancel out. This is rare and it is a real hole, and the mitigation is that the diff is a filter on what the agent sees rather than on what a human reviewing the patch sees. Merge gates should read the absolute state; the loop reads the delta.

The last thing baselining does not do is tell you whether the check was worth running. Attribution improves the signal from a check you have already decided to include; it says nothing about whether a two-second linter that finds eight percent of defects deserves a place ahead of a six-second type checker that finds thirty-one, and both of those percentages are numbers somebody had to produce. Where they come from, and why the second number is almost never what it appears to be, is the next chapter.