Never loaded, loaded stale, loaded and disregarded

The deploy skill has one rule that anybody outside the team knows about: run make preflight and paste its output before proposing a release. On a Thursday afternoon a release went out without it, a migration ran against a replica that was eleven minutes behind, and the postmortem asked the obvious question — why did the skill not stop this?

What the team had to answer with was a row in a table with two useful columns. The user request: ship the 4.2.1 hotfix to prod. The final assistant message: a four-paragraph summary of what had been deployed. Between those two strings, thirty-one tool calls had happened and none of them were recorded.

Three explanations fit that evidence equally well. The skill may never have been selected, in which case its text was not in the context and the agent was working without the rule. It may have been selected and injected in a version that did not contain the rule, because the rule had been added to the skill nine days earlier and the worker pool had been running a checkout from before that. Or it may have been selected, injected complete and current, and disregarded — the rule read, and the agent proceeded to the deploy anyway. Each of those has a different fix, and each fix is useless against the other two. Rewriting the description does nothing for a stale checkout. Redeploying the workers does nothing for an agent that read the rule and skipped it.

Three fields, and what each pair needs#

The instrumentation that separates them is three fields recorded per run.

The selection decision is the list of skills the runtime chose for this request, ideally with whatever ranking or score it used and the full candidate set it chose from. This distinguishes "never loaded" from everything else, and the candidate set matters as much as the choice, because a skill that was not even a candidate is a different problem from one that was ranked fourth.

The injected content identity is what actually entered the context: a content hash of the rendered body — a short fingerprint computed from the text itself, so that it changes if any character of the body changes — along with its length in tokens and the version identifier it came from. This is the only field that separates "loaded stale" from "loaded and disregarded", and it is the one most often left out, because the skill is in the repository and it feels self-evident which version ran.

The resulting tool calls are the ordered list of calls the agent made, with names, the relevant arguments, and exit status. This is what establishes whether the mandated behaviour happened, and it is the same trace the compliance checks in the previous chapter assert against.

The bottom-right branch is worth pointing out, because it is the outcome nobody predicts and it happened on the Thursday in question. Once the fields were in place and the run was replayed, make preflight appeared in the tool calls at position 14 with exit code 2, and the agent had read the failure, decided it was a flake in an unrelated integration test, and carried on. That is neither a trigger failure nor an instruction-following failure in the usual sense; it is a rule that says what to run and not what to do when it fails, and no amount of work on the description or the body's placement would have surfaced it.

Read down the first two rows and the point of the exercise is visible: the two fields most systems already log are the two that are identical across all three failures.

What the record looks like#

{
  "run_id": "r-8814f2",
  "ts": "2024-06-13T14:41:07Z",
  "request": "ship the 4.2.1 hotfix to prod",
  "skill_selection": {
    "candidates": ["deploy-release", "db-migrations", "incident-comms"],
    "selected": ["deploy-release"],
    "selector": "description-match@v3"
  },
  "skills_injected": [
    {
      "name": "deploy-release",
      "version": "git:4f1c9ab",
      "body_sha256": "9c1e...b70a",
      "tokens": 1840,
      "position": "system"
    }
  ],
  "tool_calls": [
    {"i": 14, "name": "bash", "command": "make preflight", "exit": 2, "ms": 51200},
    {"i": 15, "name": "bash", "command": "git tag -a v4.2.1", "exit": 0, "ms": 210},
    {"i": 16, "name": "bash", "command": "./deploy.sh prod v4.2.1", "exit": 0, "ms": 94100}
  ]
}

Storing the body itself in every record is the mistake to avoid. At roughly 8.5 KB of rendered text per skill and 1.4 skills per run across 12,000 runs a month, that is about 143 MB monthly of near-identical text. The hash costs 64 bytes and the bodies are stored once each under their own hash, which is what content-addressing buys — the repository has produced 41 distinct versions of deploy-release in its lifetime, which is 350 KB stored forever. The full record above is 2.1 KB, so the whole instrumentation layer runs at about 25 MB a month, which no one will ever ask you to justify.

The distribution changes where the work goes#

Once the fields existed, the team went back through every complaint of the form "the skill did not work" filed over the following month — 47 of them — and bucketed each by the leaf of that diagram it landed on.

SELECT CASE
         WHEN NOT skill_selected           THEN 'never loaded'
         WHEN injected_sha <> current_sha  THEN 'loaded stale'
         WHEN NOT mandated_call_present    THEN 'loaded and disregarded'
         WHEN mandated_call_exit <> 0      THEN 'ran and failed'
       END AS bucket,
       count(*)
FROM run_records
WHERE reported_failure AND skill = 'deploy-release'
GROUP BY 1 ORDER BY 2 DESC;

Twenty-two were never loaded, seventeen were loaded and disregarded, six were stale, and two were the run-and-failed case. Before the instrumentation, the team's working theory had been that the agent was ignoring the rule, and two people had spent a fortnight rewriting the body — work aimed at the second-largest bucket while the largest one went untouched. A distribution like that is the argument for building this layer before the next round of prompt engineering rather than after it.

The six stale runs had a cause worth describing because it is structural rather than careless. Skills were baked into the worker image at build time, the image was rebuilt on a weekly schedule, and the repository was the source of truth for humans reading the skill. For up to seven days after any edit, the skill in the repository and the skill in production were different documents, and every discussion about "what the skill says" was conducted against the wrong one. The fix was to have the worker refuse to start when its bundled skill hashes are not in the release manifest — the list, published with each release, of exactly which skill versions that release was cut against — which turns a silent seven-day window into a deploy that fails loudly.

When the runtime will not tell you#

Some runtimes expose the selection decision as an event; others do not, and then the field has to be recovered. Three approaches work, in descending order of how much you should like them.

If the rendered prompt is available, hash it and diff against the known skill bodies; a substring match is enough to recover both which skill loaded and which version. If only the token count is available, the presence of a 1,840-token block is a weak but usable signal, and it degrades badly once two skills are in play. The last resort is a sentinel: a distinctive, otherwise-meaningless string in the skill body, plus an instruction that the agent echo a marker in a structured field. That works, and it has the defect that it is itself an instruction the agent can disregard, which means a missing sentinel is ambiguous between "not loaded" and "loaded and ignored the sentinel instruction" — the exact ambiguity the whole exercise is meant to remove. Use it to get started and treat it as a reason to ask for a proper selection event.

Instrumentation is not evaluation#

These three fields answer questions about individual runs. They will tell you, on Thursday afternoon, which of the four things happened. They do not tell you the rate at which any of them happens, they do not tell you whether a change improved anything, and they contain no notion of expected behaviour — nothing in the record above says the skill should have loaded, because that judgement lives in a labelled corpus and not in a log line.

The relationship runs the other way. Instrumentation makes the aggregate numbers diagnosable. An activation rate falling from 88% to 61% is a fact about routing; the selection decisions behind those runs say whether the skill was absent from the candidate set entirely or present and outranked, and those are different problems with different owners. A compliance rate falling from 93% to 71% is a fact about outcomes; the injected hashes say whether the drop coincides with a body edit, and the tool calls say whether the mandated step is being skipped or is being run and failing.

The request text is the one field with a retention question attached, since it is user input and can carry anything a user typed. The other two fields are hashes, tool names and exit codes, which carry no such problem and are the fields the triage actually depends on, so a policy of thirty days for request text and a year for the rest costs nothing in diagnostic power and removes the argument entirely.

Two habits keep the layer honest. Sample rather than record everything if volume is a concern, but sample by run and never by field, because a record with the selection decision and no tool calls cannot answer anything. And put the fields in the evaluation harness as well as in production, so that a case which fails in the suite arrives with the same three fields attached; otherwise every suite failure begins with someone re-running it by hand to work out what kind of failure it was.

There is one thing the diagram genuinely cannot resolve, and it should be said plainly rather than discovered later. "Loaded and disregarded" is a bucket rather than a cause. A rule at line 140 of a 200-line body, after four worked examples, gets disregarded for different reasons than a rule that contradicts another skill loaded in the same run, and the record shows both as the same leaf. Narrowing that further means changing the body and measuring, which is the loop the next chapter is about — and which immediately raises the question of what you are comparing against, because a number with nothing to compare it to cannot show a regression at all.