Five phrasings you wrote yourself test your own vocabulary

The db-migrations skill shipped on a Tuesday, and the check that cleared it was a scratch file with five lines in it. Its author had written the description — Use when the user asks to create or modify a database migration — then opened a fresh session and typed five requests to see whether the skill loaded: write a migration to add a column to orders, create a database migration for the new index, I need a migration that drops the legacy column, generate a migration to rename user_id to account_id, and add a migration for the audit table. All five loaded it. The pull request description said "tested, 5/5" and nobody objected.

Four weeks later somebody went looking for why migrations kept landing without the rollback section the skill mandates. They pulled every agent session from the month that had ended with a file written under db/migrations/ — 240 sessions — and checked which of them had loaded the skill. Forty-one had. The skill was firing on a shade under one request in six of the work it existed to govern, and it had passed its release check with a perfect score.

A skill is a name, a one-sentence description, and a body of instructions, packaged so that an agent runtime can put the body into the model's context partway through a session. Testing a skill means checking three separate things about it, and this topic is about measuring each of them rather than forming an impression. The first is activation, the rate at which the skill loads on the requests it was written for; the scratch file above was a five-case attempt at exactly that. The second is the reverse, whether the skill stays out of work it has nothing to do with, which matters because loading is not free — the body occupies context, and any step it mandates actually runs. The third is compliance, whether what the agent did once the skill had loaded differs in some way that only those instructions explain. All three questions arrive together the first time a skill is proposed for merge, and again on every edit to it afterwards, because a skill is prose rather than code: changing one sentence can move any of the three numbers, and it reaches a reviewer as a diff that reads as harmless whether or not the behaviour underneath it has moved.

Why five hand-written requests always pass#

Skill selection compares the request against the skill's name and description, which is the mechanism established in 12.4. The author of a description is the worst available source of test requests for it, because they wrote the description minutes earlier and are still holding its vocabulary in working memory. Every one of the five phrasings above contains the word migration. So does the description. The test was a lexical overlap check against a string the same person had just composed.

Here is what the first five sessions in the logged set actually looked like, next to the five that were used as the release check.

Four of the five real requests describe an outcome in the domain — a column, an index, a type change — and never name the mechanism that produces it. The person asking does not know or care that the repository's convention is a timestamped SQL file with an up and a down section; that is exactly the knowledge the skill was written to supply. A description written in the mechanism's vocabulary can only be matched by requests that already contain the mechanism's vocabulary, which selects for the users who least need the skill.

This is the same failure as testing a support search box by typing the phrasings from the knowledge base articles you wrote. The article is titled "Resolving TLS handshake failures on outbound webhooks" and the customer types "webhooks stopped working after Tuesday". Both describe the same event; the search only matches one of them; the engineer who wrote the article will type the first phrasing every time they check.

A corpus built from traffic, not from imagination#

The replacement is a labelled set of real requests. Sourcing it is unglamorous and takes an afternoon. Agent session logs are the primary source, because they are the actual distribution the skill will face. Beyond those, the ticket queue, the pull request titles for changes of the relevant kind, and the engineering channel where people ask each other for the same work all produce phrasings nobody on the team would have invented.

The label is a single boolean answering one question: given only this request, should the skill have fired? It is decided from the request text alone, before anyone looks at what the agent did, because reading the transcript first turns the label into a rationalisation of the outcome. Two people label independently and a third settles disagreements. On a first pass over 240 migration-adjacent sessions our two labellers disagreed on 19 of them, almost all in the same region — requests like "queries on users.email are slow" where the fix might be an index (a migration) or a query rewrite (not one). Those disagreements are not noise to be averaged away; they are the boundary of the skill, and they will matter again in the next chapter.

Raw agreement of 92% flatters the exercise, because the set skews heavily towards positives and two labellers who both say yes most of the time will agree often by accident. Corrected for that skew — which is what Cohen's kappa does, discounting the agreement two labellers would have reached by chance alone — the figure is 0.62, which is the honest one and the one worth writing at the top of the file. It constrains how finely any rate built on these labels can be read: if the ground truth itself carries that much slack, a two-point movement in activation is not a finding, and tuning a description against one is tuning against the labellers' bad afternoon.

Two hundred and forty sessions was a full month of everything that touched db/migrations/, which is small enough to label by hand in a sitting and large enough to be worth the afternoon. A team with forty thousand agent sessions a month cannot label all of them and gains little by trying; sample stratified by requester and by week, so that a single sprint in which one engineer did nothing but schema surgery does not supply a third of the corpus. Fifteen sessions per requester per week is a reasonable cap. Record the sampling rule in the same file as the cases, because a corpus whose provenance is undocumented gets quietly re-sampled by the next person and stops being comparable to anything measured before.

Store the result as a versioned file in the repository next to the skill, so that a change to the corpus shows up in a diff:

{"id": "log-0412", "request": "orders needs a deleted_at timestamp, nullable, backfilled to null", "expect_skill": true, "source": "session-log", "note": "no mechanism vocabulary"}
{"id": "log-0455", "request": "can we get an index on users.email, queries are timing out", "expect_skill": true, "source": "session-log", "note": "outcome phrased as a symptom"}
{"id": "log-0471", "request": "the status column should be an enum instead of free text", "expect_skill": true, "source": "session-log"}
{"id": "tkt-2210", "request": "split address into street/city/postcode across the customer tables", "expect_skill": true, "source": "ticket"}
{"id": "log-0503", "request": "add a migration for the audit table", "expect_skill": true, "source": "session-log", "note": "does use the word; keep it, it is real traffic"}

Keep the requests that do contain the mechanism vocabulary. They occur in real traffic and dropping them would bias the corpus in the opposite direction. The point is that they should be a minority of the set in the same proportion as they are a minority of the logs, rather than being all of it.

Measuring the thing rather than demonstrating it#

The harness replays each request and records which skills the agent selected. What "selected" means depends on the runtime; most expose it either as an event in the stream or as a tool call in the transcript, and in the worst case you can detect it by putting a distinctive sentinel string in the skill body and grepping the injected context for it. What matters is that the check reads the selection decision rather than the final answer, because an agent can produce a perfectly good migration without the skill and a bad one with it.

import json, collections

cases = [json.loads(line) for line in open("corpus/db-migrations.jsonl")]
fired, missed = 0, []

for case in cases:
    run = agent.run(case["request"], skills=["db-migrations"])
    selected = "db-migrations" in run.selected_skills
    if selected:
        fired += 1
    elif case["expect_skill"]:
        missed.append(case)

n = sum(1 for c in cases if c["expect_skill"])
print(f"activation {fired}/{n} = {fired / n:.1%}")

by_note = collections.Counter(c.get("note", "unlabelled") for c in missed)
for note, count in by_note.most_common():
    print(f"  {count:3d}  {note}")

Run against the 240-case set, the shipped description scored 41/240, or 17.1%. That number is the deliverable, not the list of examples that worked. A rate is comparable across edits, it has an obvious direction, and it exposes its own uncertainty: with 240 cases at 17%, the 95% interval is roughly ±4.8 points, so a rewrite that moves activation to 21% has not clearly moved anything. Sizing a set to the difference you intend to detect gets its own treatment later in this topic; for now it is enough to know that a rate carries an error bar and a demonstration does not.

One thing the replay quietly changes is position. A request pulled from the logs was frequently turn seven of a session in which the agent had already read three files and written one, and the router that saw that request saw all of the preceding context along with it. Replaying the same sentence as the first turn of a clean session is a different decision problem, and on this set it moved activation about three points in the optimistic direction, because a bare request has less competing material around it. Either replay the preceding turns as well, which costs more and earns it for skills whose triggers are mid-task, or restrict the corpus to session-opening requests and write the restriction down where the rate is reported.

The grouped misses are what tells you how to fix it. Of the 199 misses, 143 phrased the request as a schema outcome with no mechanism word, 31 described a performance symptom whose fix happened to be a schema change, and 25 named the repository's own conventions ("add a file under db/migrations") in ways the description did not anticipate. That distribution says the description needs the domain vocabulary — column, index, table, schema, enum, backfill — and not a more emphatic version of the sentence it already has.

What an activation rate does not tell you#

Activation measures the routing decision and nothing else. A skill that fires on every request it should and is then ignored by the model scores 100% here, which is the entire subject of chapter three. Treating activation as a proxy for the skill working is the most common way this measurement gets misused, usually because it is the easiest of the three numbers to collect.

The rate is also a property of an environment rather than of a description in isolation. The same description in a repository with eleven other skills, two of which mention databases, will score differently from the same description alone, and it can move without anyone editing anything. Chapter five deals with that directly.

The measurement in this chapter has a sharper limitation, and it is structural rather than a matter of care. Every case in the corpus is labelled expect_skill: true, which means the only way the number can move is up, and the only way to move it up is to make the description match more requests. A description reading Use for any request that mentions data would score close to 100% on this corpus. That is not a hypothetical failure mode; it is the natural endpoint of iterating against a set built the way this one was built, and it is what the next chapter is for.