The removal arm is what proves worse-than-nothing

The tech lead's question took twenty-five minutes and about four dollars to answer. Sixteen agent sessions, split into two arms in the sense a drug trial uses the word: eight runs with the index line present, eight with it deleted, and every other input held identical between the two groups — the same repository commit e4b1c77, the same pinned model build, the same tool set, the same request typed the same way. The two config files differed by one line and 214 bytes of nothing else.

--- a/services/events/AGENTS.md
+++ b/services/events/AGENTS.md
@@ -12,8 +12,6 @@
 ## Querying events

-The events table has no index on `customer_id`; never filter by it directly.
-Restrict on `created_at` (indexed) and filter customers in application code.
-
 Timestamps are stored in UTC. `occurred_at` is client-supplied and may be
 skewed; prefer `created_at` for time-window queries.

The results settled the argument in a way that no amount of further discussion of the July session would have.

The two arms answer different questions than one arm does#

Chapter 3's chain establishes that in one session the line was present, cited and acted upon. Chapter 1's argument that the agent would have gone looking without it is a counterfactual, and a counterfactual asserted in prose is something a reader either accepts or does not. Running the second arm converts it into an observation. Nobody has to grant that a hypothetical agent would have read the schema; eight of eight actually did, and the trace says so:

[14:02:11] assistant (thinking): I need the last 30 days of events for one
            customer. Before writing the query I should check what indexes
            exist so the filter is sargable.
[14:02:14] tool_call: sql("SELECT indexname, indexdef FROM pg_indexes
                          WHERE tablename = 'events'")
[14:02:15] tool_result: events_pkey | events_created_at_idx |
                        events_customer_id_idx (customer_id, created_at)
[14:02:29] tool_call: write_file("services/events/api/handlers.py", ...)

The single row in arm A's "consulted the schema" line is worth reading closely, because it is the exception that shows the mechanism rather than a stray data point. In that run the agent did query pg_indexes, saw events_customer_id_idx in the output, and wrote in its reasoning that the index appeared to exist but that AGENTS.md was explicit, so the index was probably new, unusable, or about to be dropped, and it would follow the documented guidance. It then wrote the date-range fetch. The false constraint did not merely prevent a lookup; on the one occasion the lookup happened anyway, the constraint overrode the evidence. That is the sharpest form of the worse-than-nothing claim, and it exists in the record only because the arm was run eight times instead of once.

The two rows in arm B that started wrong and ended right are the other half of the finding, and they are the ones that make "worse than nothing" precise. In both, the agent wrote the naive date-range query first, ran it against the development database, saw a 6-second response and 2.9 million rows, and went back to look at the schema. Missing guidance left it wrong for ninety seconds with a recovery path open. The false constraint left it wrong permanently, because it had already been told what the slow response meant. What separates the arms is not the quality of the first attempt so much as whether the system's own feedback could still reach the agent afterwards.

Holding everything else still#

Taking one component out of a system and re-running to see what changes is an ablation, a term borrowed from experimental work on models themselves, and an ablation is only as good as its controls. There are four that get skipped and one that gets confused with rigour.

The model must be pinned to a specific build for both arms. Running arm A on Monday against whatever the provider was serving and arm B on Thursday after a silent point upgrade produces a difference you cannot attribute, and provider aliases do move underneath you. The repository commit must be identical, which sounds obvious and is routinely violated because people run the removal arm on their working branch, which by then contains the fix. The tool set must match, including MCP servers, since arm B gaining a database tool that arm A did not have would explain the whole result on its own. The request must be byte-identical, from a file rather than retyped.

The edit itself is a control too. Delete the offending lines and change nothing else, because rewriting the surrounding section, reordering entries or trimming the file "while you are in there" turns a one-variable experiment into a comparison of two documents. If you want to test a replacement wording, that is a third arm, run separately.

#!/usr/bin/env bash
# ablate.sh — one config line, two arms, n runs each
set -euo pipefail
N=8
BASE=e4b1c77
MODEL=claude-opus-4-6-20260615
TASK=./tasks/customer-events-endpoint.txt

for arm in with without; do
  for i in $(seq 1 "$N"); do
    wt="/tmp/ablate/$arm-$i"
    git worktree add -q --detach "$wt" "$BASE"
    [ "$arm" = without ] && \
      git -C "$wt" apply ../../patches/remove-index-line.patch
    agent run --model "$MODEL" --no-update \
              --allowed-tools sql,read_file,write_file \
              --prompt-file "$TASK" --cwd "$wt" \
              --record "results/$arm-$i.json"
    git worktree remove --force "$wt"
  done
done

A tempting shortcut is to ablate a whole section rather than a line, on the grounds that it is faster to delete the four entries under ## Querying events than to argue about which of them matters. Section-level ablation answers a coarser question and leaves you unable to act on the answer: if behaviour improves you know something in those four lines is harmful, the three innocent entries get deleted along with the guilty one, and the person who wrote them will reasonably object. Line-level costs the same money and produces a result you can merge.

The thing that looks like rigour and is not is turning the sampling temperature — the knob controlling how much randomness there is in the model's choice of each next token — down to zero and running each arm once. A single deterministic sample tells you what one draw from the distribution looks like, and the distribution is the object under study; a rule that changes behaviour in six runs out of eight is a different finding from one that changes it in eight, and both are worth knowing. Six to ten runs per arm is enough to distinguish "always" from "usually" from "occasionally", which is the resolution these decisions need. At roughly 48,000 input and 6,000 output tokens per run, sixteen runs cost about four dollars and, with four running concurrently, finish inside ten minutes.

The same design applies to chapter 1's export, and it is worth noting how much cheaper the answer is there than the two weeks of duplicate-row investigation that preceded it. Arm A is the repository as it stood, with the claim that the search API cannot page; arm B removes those two sentences. The task is the original ticket. What the arms produce is not a subtle difference in query quality: arm B fetches search.internal.example.com/v2/openapi.json within the first three tool calls in every run, because an unexplained 504 on an HTTP API sends an agent to the API surface. The pair is the whole argument, and it takes an afternoon less than the boundary-condition bug hunt did.

Reading a null result honestly#

Sometimes both arms behave the same, and that outcome has two very different explanations which you have to separate before drawing a conclusion.

The first is that the line is inert. The agent would have done the same thing with or without it, because the behaviour is well-established in the model, or because the surrounding code already demonstrates the pattern, or because the tool refuses the wrong action anyway. An inert line is not harmless — it occupies budget and it will be cited in a future review as precedent — but it is not the emergency that a false constraint is, and the argument for removing it belongs to the budget discussion in chapter 6 rather than to this one.

The second explanation is that your task did not exercise the rule. A rule about migration rollbacks will not show a difference on a task that touches no migrations, and the null result says nothing except that you chose the wrong probe. Distinguishing the two takes one more piece of evidence: search both arms' traces for the rule's subject. If neither arm ever mentioned indexes, migrations, or whatever the rule governs, the experiment did not run.

There is a third result that catches people out, which is that arm B is worse. Removing a line can degrade behaviour even when the line is factually stale, because the rule was doing a second job nobody wrote down — a badly-worded entry about indexes may have been the only thing telling the agent that this table is large. That is a genuine finding and the right response is to replace rather than delete, with the replacement tested as its own arm.

What the pair does not give you#

The comparison is scoped to one task on one commit with one model. It licenses a claim about this rule on tasks of this shape, and it does not license a claim about how the file behaves in general, which is why a single ablation is a decision tool rather than a quality programme. The step beyond it is to keep the task file, the two config variants and the expected outcome as a checked-in artefact, so the question can be re-asked automatically when the model changes or when someone proposes putting the line back. Treating standing instructions as things you can run tests against is the argument in Your AGENTS.md file isn't the problem. Your lack of evals is, and the ablation you have just run is the smallest possible instance of it: one task, two arms, an expected outcome.

What you hand the reviewer is also smaller than what you collected. The pull request that finally removed the index line carried the one-line diff, the results table above, and two trace excerpts of about fifteen lines each — the arm A run that saw the index and deferred to the file, and any arm B run. Sixteen full session records exist in results/ for anyone who wants them, and nobody asked. A reviewer needs the comparison to be checkable rather than exhaustive, and a page they will read beats an archive they will not.

Cost is the honest limitation. Four dollars and twenty-five minutes is nothing for a line that has been generating slow endpoints for seven months, and it is far too much for each of the sixty entries in a file you inherited yesterday. Ablation is what you spend on a suspect that has already been narrowed down by the cheap checks in chapter 2, and on entries whose removal somebody is resisting. Most stale lines never need it, because nothing about them is in dispute once the pg_indexes output is on the table.

The deeper limitation is that all of this is retrospective. The index landed in January, the endpoint shipped in July, and the ablation ran in August — seven months in which every session on that service inherited a false constraint, discovered eventually because a compliance report was slow enough to be noticed. Nothing in chapters 2 through 4 shortens that interval; they are instruments for a fault that has already been suspected. Shortening it means arranging for the migration that added the index to be the thing that flags the config file, which turns review from a calendar activity into a consequence of the change that invalidated the rule.