Moving guidance into a skill replaces a certainty with a match

The infrastructure team's CLAUDE.md carried a 400-line release runbook — a runbook being the written procedure a team follows to carry out one operational job, here everything from cutting the branch to rolling the deploy back — about 3,100 tokens by the tokenizer, sitting inside a file of 4,900. In March someone moved it into a skill. The prompt-token graph in their dashboard stepped down from a median of 5,400 tokens per session to 2,300 the following Monday and stayed there. Nobody complained, which is the outcome everyone treats as confirmation. Sessions got marginally cheaper, the file got readable again, and the change appeared in the quarterly write-up as a win.

In May a release went wrong. A migration that adds a non-null column without a default went out ahead of the deploy that populates it, the write path started rejecting inserts, and the on-call spent forty minutes finding it. The runbook covered exactly that: it had a section called migrations that expand before they contract, with the two-phase ordering and the check to run before merging. The agent that wrote and merged the migration never read it. The skill did not load on that session, and nothing anywhere recorded that it had not.

A description is a query surface, not a label#

Moving content into a skill changes what determines whether the agent sees it. In the config file the content was unconditionally present; every session got it whether or not it applied. In a skill the content is present when the skill's description matches the request, and the description is the entire routing surface — a couple of lines of text, matched against whatever the user happened to type.

---
name: release-runbook
description: >
  The team's release process: cut a release branch, run the smoke suite,
  the approval chain, changelog entries, and the rollback drill.
---

## Cutting a release
...

That description is a fair summary of the document and a poor query. It reads like a table of contents entry, which is what it was written as, because the person who moved the content had the document in front of them and wrote a sentence describing it. What decides whether the skill fires is not whether the description is accurate; it is whether the words in it overlap with the words in the request. The migration request that went wrong in May read: add a tenant_id column to subscriptions and backfill it. There is no release in that sentence, no smoke suite, no rollback. The document covered the task; the description did not.

This is the same problem an on-call runbook wiki has, and teams already know its shape from that context. Runbooks indexed by what the procedure is called are unfindable at three in the morning, because the person searching does not know the procedure's name; they know the symptom. Runbooks indexed by symptom get found. A skill description is the search index, with the difference that the searcher never tells you it came up empty.

Measuring whether the match holds#

The check the March move skipped is not complicated, and it takes an afternoon. Collect the requests that the always-loaded text used to cover, run them, and record whether the skill loaded. The requests come from the same place chapter 1's reach estimates came from — the opening turns of real sessions — filtered to the ones the runbook was relevant to, which for this team was 40 sessions over the preceding quarter.

#!/usr/bin/env bash
# scripts/skill-fires.sh — replay recorded first turns, record skill loads.
set -euo pipefail

SKILL="release-runbook"
fired=0 total=0

while IFS= read -r prompt; do
  total=$((total + 1))
  transcript=$(claude -p "$prompt" --max-turns 1 --output-format json)
  if printf '%s' "$transcript" | jq -e --arg s "$SKILL" \
       '[.. | strings] | any(contains($s))' > /dev/null; then
    fired=$((fired + 1))
    printf 'FIRED  %s\n' "$prompt"
  else
    printf 'MISSED %s\n' "$prompt"
  fi
done < prompts/release-related.txt

printf '\n%d/%d fired (%d%%)\n' "$fired" "$total" $((fired * 100 / total))

For this skill the answer came back 27 of 40, or 68%. The interesting part is not the percentage; it is that the misses cluster, and reading the thirteen of them takes two minutes.

Before reading the causes it is worth being clear about what that 68% is a measurement of, because it is easy to over-read. Forty replayed prompts put a fairly wide interval around the true rate — roughly ten percentage points either side at ordinary confidence — so 68% and 62% are the same finding and there is no point tuning against the difference. The number is also specific to the requests that actually occurred in one quarter on one repository; it says nothing about phrasings the team has not yet used. And replaying only the first turn understates firing, because a skill can also load later in a session once the agent has read enough of the codebase to name what it is doing. None of that undermines the check. A rate in the sixties is far enough from certainty that the direction of the finding survives every one of those caveats, and the misses are individually readable, which is where the actual value is.

Two distinct causes are visible there. The first three misses are vocabulary: the description uses the team's formal words for the process and the requests use the words people type under pressure. That is fixable by writing the description as an index of situations rather than as a summary of contents — naming ship, deploy, prod, hotfix, patch, roll back, on-call, and the failure vocabulary alongside the process vocabulary. Re-running the same 40 prompts after that edit took the team to 35 of 40.

The last cause is not fixable by editing the description, and it is the one that caused the May incident. The migration ordering rule was in the runbook because the person who wrote the runbook thought of migrations as part of releasing. The people who write migrations do not think of them that way; they think of them as ordinary schema work. A rule filed under a workflow the person doing the work does not believe they are in will not be retrieved by any description of that workflow. That rule was never a runbook section; it is an invariant about how migrations must be shaped, it has reach across every task that touches the schema, and by chapter 1's test it belongs in the always-loaded file. The move was right for 380 lines of the runbook and wrong for the twenty that were never really about releasing.

Relocation is not deletion, and it is not free#

The reason this failure is hard to catch is that both mechanisms are silent in the same direction. A config entry that never changes a decision costs tokens and shows no signal. A skill that never fires costs almost nothing and shows no signal either, which is worse, because the absence of cost removes the only pressure that would have made someone look. The token graph that made the March move look like a win is measuring the thing that got better and is structurally incapable of measuring the thing that got worse.

Something else is worth being precise about: a skill firing is not the same as the agent following what the skill says. The 35 of 40 figure is the retrieval rate. On top of it sits whatever fraction of loaded-and-relevant sessions actually apply the content, which is a separate number with its own measurement. Chapters 6 and 7 deal with why that second number is never 100%. For now the point is that moving content out of the config file introduces a multiplication where there used to be a single term: the content's effect is now the firing rate times the compliance rate, and only the first of those was created by the move.

The version of this that does real damage is relocation performed to avoid an argument. Somebody proposes cutting a section; its author objects; and moving it to a skill resolves the conflict because nobody has to concede anything. The file gets smaller, the author's content survives, the meeting ends. What has happened is that a cost problem was converted into a routing problem and then declared solved, and because the purpose of the move was social rather than technical, no one runs the firing check — running it would reopen the argument the move existed to close. Six months later the file is half its old size and some unknown fraction of the moved content has quietly stopped working, with no record of which fraction.

The check itself is not free either, and pretending otherwise makes it easy to skip. Forty single-turn replays on this team's setup cost about eleven minutes of wall clock and a little under two dollars, plus the half hour spent assembling the prompt list from session logs and the twenty minutes spent reading the misses. That is cheap against a forty-minute incident, and it is not cheap enough to run casually on every skill in a repository that has thirty of them. The sensible cadence is to run it when content moves and when a skill's description is edited, and to accept that a skill nobody has ever measured is a skill whose firing rate is unknown rather than assumed good.

There is a plain test for whether a proposed move is honest. Ask what evidence would tell you the move was a mistake. A move made on the merits has an answer: a firing rate below some threshold on the requests the content used to cover, measured within a week. A move made to end an argument has no answer, because its success criterion was the absence of further discussion. If nobody is willing to own the firing check, the move is not ready, and the more truthful options are to cut the content or to keep carrying it.

What the move does not tell you#

Verifying that a skill fires establishes that the content reaches the agent on the right requests. It does not establish that the content was worth carrying anywhere, in any mechanism. The 400 lines might have been changing nothing at all — a possibility that survives every measurement in this chapter, because a firing check confirms delivery and says nothing about effect. A skill that fires on 100% of relevant requests and changes no decision is a slower, more elaborate way of achieving what deleting it would have achieved.

It also does not tell you what the section was costing while it sat in the config file. The team's dashboard said 3,100 tokens per session, and that figure is a poorer guide to real cost than it looks: whether those tokens are billed once and read from cache twenty times, or billed twenty times at full price, depends on details of how the harness assembles the prompt that nobody on the team had checked. The next argument on this file will be about a new 500-token section, and it will be conducted entirely in adjectives — short, cheap, worth it — unless somebody arrives with numbers for what a section actually costs and what it actually changes.