Some invariants belong in the runner, not in the file

At 14:02 on a Wednesday an agent session working a ticket to populate idempotency_key on historical invoices ran a backfill — a one-off job that rewrites rows already sitting in a database, to fill in a new column or correct an old one — against the production environment:

pnpm backfill invoices:idempotency --env production --apply

It rewrote 2.4 million rows. The backfill also touched updated_at on the 180,000 rows where the column was null, which put those invoices at the head of the search re-index queue and degraded invoice search for 40 minutes. Restoring the timestamps from the logical backup took three hours with six people on the call. The incident review opened with the config file, which has said since the previous October:

- Backfills must be run against staging first.

The line reads as sound practice, it was written by a competent engineer after a real near-miss, and it is present in every session. It also could not have been followed as written, and understanding why is the difference between a remediation that works and one that adds confidence on top of the failure.

Proving the instruction is stale#

Assertion is easy and worthless here, because everyone in the room already has an opinion. Three lines of evidence turn it into a finding.

The first is the mapping from the instruction's words onto the current tooling. "Run against staging first" describes a two-step sequence: point the backfill at staging, look at what it did, then point the same backfill at production. That sequence existed in October, when backfills were hand-written scripts invoked with a DATABASE_URL in the environment. In April the team shipped scripts/backfill.ts, which takes a named job, defaults --env to staging, and refuses to write anything without --apply. Under that runner, the first step of the instruction is what happens if you type the command with no flags at all. Following the instruction is indistinguishable from not thinking about it.

The second is the artifact test, which generalises past this case. Ask what a compliant run produces that a non-compliant run does not. In October the answer was concrete: the staging run's row counts, pasted into the ticket, which is why the near-miss that prompted the line was caught. Today there is no such artifact, because the staging run is the default and produces nothing anybody records. An instruction whose compliance leaves no trace cannot be verified, cannot be audited, and cannot be the subject of a meaningful incident finding, which is the operational definition of stale that matters.

The third is the trace archive. Across the last 31 sessions that ran any backfill, the word "staging" appears in the agent's reasoning zero times. That is not evidence of defiance; it is evidence that the line has not been a live consideration in any of the last 31 opportunities, which is a stronger claim than anything the incident's single trace supports.

$ git log -1 --format='%h %cs' -L'/Backfills must be run/,+1:CLAUDE.md'
7b21e5a 2024-10-08

$ git log -1 --format='%h %cs' --diff-filter=A -- scripts/backfill.ts
d0af113 2025-04-22

The same three questions sweep the rest of the file, and on this one they flagged two more entries. For each line, name the thing that implements it today; if the answer is "a person, by hand", check whether that is still true. "Deploys go out behind a feature flag" was written when flags were created through a web console and is now implemented by a decorator that fails the build without one, so the sentence describes a step nobody performs. "Ask before adding a dependency" was written when the lockfile was reviewed by one person and is now a CODEOWNERS rule on pnpm-lock.yaml, the file in which a repository names who must review changes to a given path before they can merge. Neither is dangerous, and both are the same shape as the backfill line: text describing a manual procedure that tooling absorbed, left in place because nobody re-reads a file that never fails.

The trace shows non-compliance, and that is the wrong stopping point#

The session's trace contains no mention of the instruction. The reading it invites is that the agent ignored a clear rule, which makes this a compliance failure and points remediation at wording: strengthen it, move it to the top of the file, require the agent to state that it ran against staging before proceeding. That reading stops one question short, because "the agent ignored it" presumes the instruction was followable. Given that compliance produces no artifact, a trace cannot distinguish an agent that skipped the rule from one that satisfied it vacuously by using the runner's default on some earlier invocation.

Grant the non-compliance anyway and measure what the proposed remedy buys. Twenty paired runs on a backfill fixture with the escalated wording in place — top of the file, bold, with an explicit "state the staging result before running with --apply" — produced 17 runs that declined to pass --env production and 3 that did. Twenty runs against a runner that requires an artifact from a staging run produced 20 that could not proceed, because the process exited before touching a connection.

Fifteen percent is a large number when the action is irreversible. The team runs roughly 31 backfill sessions a year, so a 15% failure rate is between four and five production incidents a year, each carrying something in the region of thirty engineer-hours plus customer-visible degradation. Escalated wording is a real improvement over the status quo and it is not a control, because a control for an unbounded loss has to have a failure probability that does not scale with how the model happens to sample on a Wednesday afternoon. The escalation also carries the leak demonstrated in the previous chapter: strengthened prohibitions generalise past their intended scope, and in six of those twenty runs the agent also declined to run the read-only --dry-run against production, which is something the team wants it to do.

Moving the guarantee out of the text#

The remedy has three pieces, and the reason there are three is that they fail in different places.

The runner acquires the precondition the instruction always presupposed. A production apply now requires a receipt from a staging run of the same job within the last 24 hours, which converts a sentence into a check and, incidentally, creates the artifact that made the instruction auditable in the first place:

// scripts/backfill.ts
if (env === "production" && apply) {
  const receipt = await loadStagingReceipt(job, { maxAgeHours: 24 });
  if (!receipt) {
    fail(
      `No staging run recorded for "${job}" in the last 24h.\n` +
      `Run: pnpm backfill ${job} --env staging --apply\n` +
      `That writes a receipt to .backfill-receipts/ which this command requires.`
    );
  }
  console.log(`Using staging receipt ${receipt.id}: ${receipt.rowsAffected} rows`);
}

The harness acquires a pre-execution guard, meaning a rule the agent tool itself applies to a proposed command before that command is allowed to run, so that the protection does not depend on the agent choosing to invoke the project's runner rather than reaching for psql directly:

{
  "permissions": {
    "deny": ["Bash(*--env production*--apply*)", "Bash(psql*production*)"],
    "ask": ["Bash(pnpm backfill*)"]
  }
}

The config file acquires one sentence, which does the job text is actually good at:

Backfills run through `pnpm backfill <job>`. Production applies require a staging
receipt from the last 24h; the runner will tell you how to produce one.

That last piece looks redundant next to two layers of enforcement, and removing it costs more than it saves. Running the fixture ten times with the guard in place and no line in the file, the agent averaged 4.2 extra turns per task discovering the precondition by trial and error, and in 2 of the 10 it concluded the runner was misconfigured and asked the user to run the backfill by hand — the silent-compensation pattern from the fourth chapter, arriving by a different route. With the sentence present, the average dropped to 0.4 extra turns. Enforcement stops the bad outcome; text is what stops the agent from wasting a session fighting the enforcement.

The fourth piece is the one that keeps the other three honest, and it is the same move as the previous two chapters: a test that reproduces the original failure. Two tests, in fact, because the failure has two halves. A unit test on the runner asserts that a production apply with no receipt exits non-zero and touches no connection, which is fast and runs on every pull request. A trace predicate over the backfill fixture asserts that the agent, given the same ticket that caused the incident, ends the session without any production write, which is slow and runs nightly. The unit test is the one that will actually catch the regression, since the guard is code and code is testable; the trace predicate is what notices if the agent starts routing around the runner altogether.

Which entries can stay as text#

The decision turns on what happens when the entry is not followed, along two axes that are easy to assess for any given line.

The bottom-right cell is the one where a standing sentence is never the remedy, because both of the things that make text workable are absent: the model may decline to attend to it, and nothing downstream will catch the result. The backfill sits there. So does anything that spends money, sends messages to customers, or mutates a system of record.

Each mechanism buys its guarantee at a different price, and the prices are not small.

The CI row deserves its own warning, because it is the mechanism teams reach for by reflex after an incident. A CI gate can stop a dangerous migration from merging; it cannot stop a command that runs inside a session from writing to a production database, because that write happens hours before anything reaches CI and leaves no artifact for CI to inspect. Reaching for a CI check against a runtime side effect produces a control that feels solid and covers nothing.

Guards have their own staleness, which is the honest limitation of everything in this chapter. The deny pattern above matches --env production; a runner that later accepts -e prod will slip past it, and the guard will keep reporting green. Guards are at least code, so they can be tested, and a test that asserts the guard blocks the current spelling of the dangerous command is cheap to write and will fail when the spelling changes. The friction cost is the one that actually kills guards in practice: a guard that fires falsely often enough acquires a bypass environment variable in CI, and six weeks later the bypass is set everywhere.

Five chapters of criteria now sit on each entry: it has to change a decision on a nameable task, fit a budget, be verified by something in the repository, be unambiguous against every other entry, and be the kind of thing text can carry at all. Applying all five yourself, on a file you wrote, is one problem. Writing them down so that a reviewer who did not write the file can apply them to somebody else's proposed line is a different one, and the obvious way to write them down turns out to be trivially satisfiable.