Testing and iterating on skills

Read these first

• Writing a skill
• Deterministic and code-based checks

What you will know

• Test that a skill triggers when it should

  Use a corpus of realistic paraphrases with expected activations, then measure missed triggers rather than relying on a few hand-picked examples.

• Test that it does not trigger when it should not

  Include neighboring tasks and adversarially similar wording so false activations are visible, since unwanted invocation can add latency or apply the wrong constraints.

• Verify the agent actually followed the instructions rather than skimming

  Assert observable behavior in outputs, tool calls, and traces; a skill appearing in context is not evidence that its requirements affected execution.

• Catch a skill regression after an edit

  Version the evaluation cases and compare results against a recorded baseline, with thresholds that expose changes in activation, compliance, and task quality.

• Instrument runs to distinguish trigger failures from instruction-following failures

  A missing invocation and an invoked-but-ignored instruction need different fixes, so logs must preserve the decision, injected content, and resulting behavior.

Chapters

1. 1

   Five phrasings you wrote yourself test your own vocabulary

   All five fire before release; in production the skill catches one request in six that should have matched. The author shares the skill's vocabulary, so hand-written examples measure nothing about the requests real users send. Build the positive corpus from logged requests, labelled for whether the skill should have fired, and report missed triggers as a rate rather than a set of demonstrations that worked.

   Read chapter
2. 2

   A suite of positives can only be improved by matching more

   Sixty requests that should all trigger, at 93% activation, is structurally blind to the skill firing on neighbouring work and imposing its constraints and latency where nobody wanted them. Any description change that broadens matching scores well on such a suite, which is why 96% to 99% is not by itself a reason to ship. Add negatives: adjacent tasks and adversarially similar wording.

   Read chapter
3. 3

   Compliance is a difference only the instructions explain

   A model asked whether a transcript shows the agent following the rule is reading a narrative about the work, not the work, so it will report 100% while three migrations shipped without a rollback section. Assert on the artefact and the tool calls instead: the file contains a rollback section, the check command ran. Know which claims survive as deterministic checks and which genuinely need grading.

   Read chapter
4. 4

   Never loaded, loaded stale, loaded and disregarded

   A reviewer reports the pre-deploy check was skipped and your logs hold the user request and the final message, which cannot tell those three apart. Record the selection decision, the exact body text injected, and the tool calls that followed; each pair of failures needs a different fix, and only the three together separate them. Missed triggers and ignored instructions look identical from the output.

   Read chapter
5. 5

   Activation up, compliance down: the edit that scored well

   Rewording a description takes activation from 40% to 85%, and a week later the central rule is violated more often, because the skill now matches requests its body was never written for. A baseline has to record activation, compliance and task quality together and be compared after every edit. The same instrument explains a skill whose activation falls from 88% to 61% with no edit at all, and separates the two candidates: re-run the frozen labelled set, and a drop there means a newly added neighbour is competing, while a held number means the production request mix moved rather than the routing.

   Read chapter
6. 6

   Three points on thirty cases is not a difference

   A baseline is only actionable once you know which movements mean anything, and this material hands the reader small percentages on stochastic runs. With about one run in twelve failing for reasons nobody can reproduce, 96% against 99% on thirty requests is inside the noise, and shipping on it is exactly how a positives-only suite gets quietly broadened. Size the labelled set to the difference you care about, reuse the same cases so every comparison is paired, repeat each case enough times to see the run-to-run spread, and write down the delta on activation, compliance and task quality that trips the baseline.

   Read chapter
7. 7

   A 25-minute suite that is red one run in twelve gets switched off

   Predict what happens to a 200-invocation evaluation in CI within two months, because that prediction determines the design. Split it: a fast subset of activation and deterministic compliance checks per pull request, the full run nightly against a recorded baseline. Retrying failures until one passes converts a real signal into noise, and zero temperature does not make an agent loop deterministic — tool results, timing and retries reintroduce variance.

   Read chapter