Proving a rule stale takes current evidence, not memory

At 16:20 on a Thursday an engineer asked the agent to promote checkout-api v3.11.2 from staging to production. The agent read the release ticket, read AGENTS.md, and stopped before doing anything, replying that promotion requires a manual smoke test against staging first and that it could not proceed until someone confirmed the smoke test had passed. The engineer, who had promoted a dozen builds that month without ever running a smoke test by hand, told it to go ahead. The agent asked again, in more detail, quoting the rule. Then it tried to be helpful: it went looking for the smoke suite, found scripts/smoke.sh in the repository, ran it against the staging URL, and got 14 failures out of 31 checks, because that script had been superseded eleven months earlier and still asserted against response shapes from the v2 API. The agent reported the failures as a release blocker. The promotion happened forty minutes later, by hand, by an engineer who had lost confidence in her own release process for the length of a coffee break.

The line in the config file read:

## Releases

Deploys require a manual smoke test against staging before promotion.
Run `scripts/smoke.sh $STAGING_URL` and confirm all checks pass.

Everyone on the team, asked afterwards, agreed that this was "probably out of date". That is where most of these investigations begin and, unfortunately, where a great many of them end — with a shared shrug and a line that stays in the file, because nobody wants to be the person who deleted the release safety rule on a hunch. Chapter 1 left the question of how you convert a suspicion into something a reviewer can act on. The answer is that you go and look at the system that the rule describes, and you keep what you find.

Three artefacts settle it#

A claim about a rule is a claim about the world in the present tense, so the evidence has to come from the present-tense system. For the smoke test rule that means three things, gathered in about twenty minutes.

The first is the current definition of the thing the rule governs. Promotion is a pipeline job, and the pipeline is a file:

# .github/workflows/promote.yml
name: promote
on:
  workflow_dispatch:
    inputs: { version: { required: true } }

jobs:
  smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pnpm install --frozen-lockfile
      - run: pnpm test:smoke --target=https://checkout.staging.internal
        # 31 checks, ~2m10s

  promote:
    needs: smoke          # promotion cannot start unless smoke passed
    runs-on: ubuntu-latest
    steps:
      - run: ./bin/promote --service checkout-api --version ${{ inputs.version }}

The smoke suite runs on every promotion, and needs: smoke means a failing suite blocks the promote job outright. The rule asks a human to do something the pipeline already does and will not proceed without.

The second artefact is when that became true and who made it so, because a reviewer's next question is whether you are looking at a recent, possibly temporary state.

$ git log --diff-filter=A --format='%h %ad %an %s' --date=short \
    -- .github/workflows/promote.yml
9c41ae2 2025-09-18 priya.raman  ci: promotion runs smoke suite as a gate (#2214)

$ git log -S 'scripts/smoke.sh' --format='%h %ad %s' --date=short
9c41ae2 2025-09-18 ci: promotion runs smoke suite as a gate (#2214)
1f80b3d 2023-04-02 ops: add manual staging smoke script (#611)

$ gh run list --workflow promote.yml --limit 200 \
    --json conclusion --jq 'group_by(.conclusion)|map({(.[0].conclusion):length})'
[{"success":188},{"failure":11},{"cancelled":1}]

The gate landed on 2025-09-18 in PR #2214, and 200 promotions have run through it since, so this is not a state that appeared last week. The same search shows scripts/smoke.sh has not been referenced by anything in the repository since that commit, which explains why it rotted quietly against the v2 API for eleven months.

The third artefact is a replay in which somebody follows the instruction literally and finds that it names something which no longer exists in usable form. That is what the Thursday session already was, and it is worth keeping the transcript rather than describing it, because the transcript shows the rule producing the failure rather than merely coexisting with it. The instruction says run scripts/smoke.sh; running it produces 14 failures against assertions for an API version that was retired; the failures are not a signal about the build. A step that cannot be performed correctly is a step that has expired, whatever its wording says.

Those three together — definition, date, replay — are a case that survives being handed to someone who was not in the room. None of them require the reader to trust your memory or your seniority, which is the property that matters, because the person reviewing the deletion is usually not going to re-derive it themselves.

Evidence sources are not interchangeable#

Teams reach for whatever is nearest, and what is nearest is usually the weakest thing available. It is worth being explicit about what each source can establish.

The bottom three rows are the ones that produce artefacts. The top three produce opinions, and the second row produces opinions with a false air of consensus.

Polling colleagues fails in both directions, which is what makes it so misleading. When the release rule was circulated in the team channel, 11 of 14 respondents said yes, that is required. Not one of them had run scripts/smoke.sh in the previous year. They affirmed it because it describes something a careful team plausibly does, and disagreeing would have meant claiming that releases go out without a smoke test, which nobody wanted to be on record saying. The same poll, run over the whole file, produced no recognition at all for a genuinely load-bearing line about a service boundary, because that boundary had held for two years and therefore never came up in anybody's week. A survey of recall gives you the file's reputation. The property you need is whether each line is currently true.

The fourth row deserves its caveat spelled out, because a configuration file is a statement of intent and the thing that runs is sometimes a different object. A pipeline definition can be overridden by a repository setting, a required check can be marked non-blocking at the branch-protection layer, an environment can carry a SMOKE_SKIP=1 variable that nobody remembers setting. The defence is to pair the definition with a small amount of execution history, which is why the gh run list output belongs alongside the YAML: 188 successful promotions is a claim about what happened, and no branch-protection setting can make that number a fiction. Where history is thin — a promotion path used twice a year, a migration guard that has never fired — the definition is all you have, and the honest form of the finding says so rather than overstating its confidence.

This is also the flaw sitting inside an annual attestation, where each team confirms that its config file is accurate. Attestation is the first row of the table performed at organisational scale and countersigned. It can reach 100% compliance while the file contains confidently wrong lines, because confirming from memory is exactly the evidence source that the wrong lines survived the first time. What to build instead is chapter 5's subject; the point here is that the audit was never measuring what its number implied.

The same procedure against a database claim#

Change the domain and the shape holds. Take the rule that the events table has no index on customer_id and that queries must never filter by it directly, which has been steering the agent into pulling whole date ranges and filtering in application code. The current definition lives in the catalogue rather than in a YAML file:

-- current definition
SELECT indexname, indexdef FROM pg_indexes
WHERE tablename = 'events';

--        indexname         |                        indexdef
-- -------------------------+---------------------------------------------------------
--  events_pkey             | CREATE UNIQUE INDEX ... USING btree (id)
--  events_created_at_idx   | CREATE INDEX ... USING btree (created_at)
--  events_customer_id_idx  | CREATE INDEX ... USING btree (customer_id, created_at)

The date comes from the migration that added it — migrations/20250114_add_events_customer_idx.sql, merged on 2025-01-14, which is where the "seven months ago" in everyone's vague recollection turns into a fact anyone can check. The replay is the query the rule forbids, run twice under EXPLAIN (ANALYZE, BUFFERS), which executes the statement and reports the plan the database actually chose along with the rows and time at each step:

EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM events
WHERE customer_id = 8814 AND created_at >= now() - interval '30 days';

-- Index Scan using events_customer_id_idx  (actual time=0.041..3.9 rows=1,204)
--   Buffers: shared hit=38 read=11
-- Planning Time: 0.19 ms
-- Execution Time: 4.1 ms

-- The rule's prescribed alternative: pull the date range, filter in app code
SELECT * FROM events WHERE created_at >= now() - interval '30 days';
-- Seq Scan on events  (actual time=0.02..6412 rows=2,930,517)
-- Execution Time: 6,470 ms  — 2.9M rows to the client, ~740 MB over the wire

Four milliseconds against six and a half seconds, 1,204 rows against 2.9 million. The rule is not merely out of date; following it is now about 1,500 times slower than the thing it prohibits, and the measurement is one command long. Whoever reviews the deletion does not need to know anything about this service to read that output, and can re-run it in thirty seconds if they doubt you.

Reading a file you know nothing about#

The smoke-test and index cases both started from a specific suspicion. The harder version is a 4,000-token config file for a service you have never worked on, with no owners and no dates and two days to spend. Domain knowledge is what you do not have, so sort the file by what can be checked without it.

Most entries name something concrete: a path, a command, a table, an endpoint, an environment variable, a service, a flag. Those names either resolve in the current system or they do not, and resolving them is mechanical. A file of that size runs to roughly 55 to 70 entries; on the last one I worked through, 23 named something greppable, and of those, 9 named a path or script that no longer existed anywhere in the repository. That is nine entries proven stale before lunch on the first day, with a one-line artefact each, and no understanding of the domain required. The rest split into entries whose subject exists but whose claim about it needs a replay — the index case is this shape — and entries that name nothing checkable at all, such as preferences about style and instructions to be careful. The third group cannot be verified because it does not assert anything falsifiable, which is itself the finding.

Order the two days by that split rather than by reading the file top to bottom, since the cheap checks retire the largest fraction of the file per hour and give you the vocabulary to attack the expensive ones. There is a second reason to start with the greppable entries when you are new to a service: each one you check teaches you where things live. Resolving a claim about services/ledger/reconcile.py requires finding out that reconciliation moved into a worker two years ago, and by the time you have chased down nine of those, you have assembled a rough map of the system from the outside, which is precisely the domain knowledge you started the exercise without. The tooling can help here as well — mdarena scores a config file against the repository's own pull request history, which is a fast way to find entries that describe work nobody in this repository has done for a long time.

Along the way, record the command beside each finding. Evidence has an expiry of its own, and a note saying "verified 2026-08-09 with SELECT indexname FROM pg_indexes ..." can be re-run by the next person, whereas "verified, stale" cannot.

What proving staleness does not prove#

A stale line and a harmful line are different claims, and the evidence above only establishes the first. You have shown that scripts/smoke.sh no longer works and that the pipeline gates on its replacement. You have not shown that the config line is what made Thursday go badly, and that distinction is exactly where these investigations get stuck, because the reply you will get is that the agent is unreliable in general and would probably have got confused about the release anyway. That reply is not unreasonable on its face; agents do produce variable output, and one awkward session is thin material for a causal claim.

Closing it takes a different kind of artefact from the ones in this chapter: not evidence about the world, but evidence about the session — what was asked, what configuration was actually assembled into the context, what the agent said while deciding, and what it did. That chain is the subject of the next chapter.