A suite of positives can only be improved by matching more

The rewrite landed the following week. The description picked up the domain vocabulary the grouped misses had asked for — column, index, table, constraint, enum, backfill, schema change — and activation on the 240-case corpus went from 17.1% to 79.2%. The pull request showed both numbers, the corpus file was in the diff, and the review took four minutes.

Eleven days later a platform engineer opened a session and asked why the orders list endpoint was doing a sequential scan on users. The answer was a query rewrite: an ORM call was fetching a relation it did not need, and the scan disappeared once the select_related was corrected. The agent did not get there. It loaded db-migrations, which opens with a mandatory step — run make db-verify against a scratch database before proposing any change — and that job takes four minutes and eleven seconds on the team's runner. Then, because the skill's body is written entirely in terms of producing a migration, the agent produced one: a new index on users.id, with a correctly formatted rollback section, solving a problem that did not exist. The engineer spent twenty minutes before noticing the agent had been handed a different job than the one they asked about.

One degree of freedom points in one direction#

The corpus from the previous chapter has 240 cases and every one of them is labelled expect_skill: true. Think about what that means for any edit to the description. Broadening the description — adding words, loosening a condition, dropping a qualifier — can only leave a positive case matched or turn an unmatched one into a match. Narrowing can only leave a case matched or unmatch it. The score is therefore monotone in how permissive the description is, meaning it can only move one way as the description loosens, and the optimisation has exactly one direction that ever looks like progress.

Follow that gradient and you arrive somewhere absurd. A description reading Use for any request involving data, storage, or the database would have scored somewhere near 97% on the 240-case set, and it would be an unambiguous improvement by the only measurement in the room. The team that shipped the 79.2% rewrite was one more iteration away from that, and nothing in their suite would have objected.

This is why the jump from 96% to 99% on a positives-only suite is not, on its own, a reason to ship anything. The improvement is real in the sense that more of the intended requests now match. It is also unfalsifiable in the direction that matters, because the suite has no case that a broader description could fail. Any change that scores well on it might have bought those three points by matching thirty unrelated requests, and the instrument cannot tell those two stories apart. Reviewers reading "activation 96% → 99%" are reading a number that is compatible with the change being excellent and with it being a disaster.

What a false activation actually costs#

The reason to care is that an unwanted invocation is not free, and its cost is specific to the skill rather than generic. Three things happen when a skill loads on work it was not written for: its body occupies context, its mandated steps execute, and its constraints shape the output.

For the second row of that table, false activation is a rounding error and the right operating point is close to "fire whenever plausible". For the first, a false activation costs four minutes of wall clock, a lock on a shared resource, and an answer aimed at the wrong problem. Choosing a threshold without knowing which of those two situations you are in is how a skill with a permissive description ends up quietly taxing every database-adjacent request in the repository.

The number that made this concrete for the migrations team came from widening the log pull. The 240-case corpus had been built from sessions that ended in a file under db/migrations/, which by construction contains almost no negatives. Re-running the pull over every agent session in the same month gave 1,100 sessions, of which 860 were not migration work, and the rewritten description fired on 214 of them — a false activation rate of 24.9%, the share of requests the skill had no business loading on where it loaded anyway. One database-adjacent request in four was now paying the four-minute verify step for nothing.

Where negatives come from#

A negative is a case labelled expect_skill: false — a request the skill is supposed to sit out — and the positives are the expect_skill: true cases the previous chapter assembled. Negatives are harder to source, because nobody logs the requests a skill correctly ignored; they look like ordinary work. Four sources between them cover the space, and they are not interchangeable.

The broadened log pull is the base layer, and it is the only one that reflects real proportions. Every session in the repository, labelled with the same boolean and the same two-labeller protocol, gives a negative set whose composition matches production. It is also the least efficient source per unit of labelling effort, because most of what it returns is obviously irrelevant — a request to rename a React component is a negative for db-migrations in a way that teaches nothing.

Sibling skills supply the interesting middle. Every other skill in the repository has, or should have, its own positive corpus, and each of those corpora is a negative corpus for this skill. When db-performance was added, its 90 positives went straight into db-migrations's negative set, and that pairing is what surfaces genuine competition rather than hypothetical overlap.

Adversarial near-misses are hand-written, and this is the one place where writing your own requests is the right move, because you are deliberately probing the boundary rather than sampling the distribution. Take a positive and make the smallest edit that flips the label:

{"id": "adv-01", "request": "add an index on users.email", "expect_skill": true}
{"id": "adv-02", "request": "explain why there's an index on users.email", "expect_skill": false, "derived_from": "adv-01"}
{"id": "adv-03", "request": "the status column should be an enum instead of free text", "expect_skill": true}
{"id": "adv-04", "request": "what would break if status became an enum instead of free text", "expect_skill": false, "derived_from": "adv-03"}
{"id": "adv-05", "request": "migrate the customer data to the new billing vendor", "expect_skill": false, "note": "uses the word, means something else"}
{"id": "adv-06", "request": "the orders table is missing a created_at index in staging only", "expect_skill": false, "note": "environment drift, not a schema change"}

Case adv-05 is the shape that catches lexical descriptions: it contains the strongest trigger word in the vocabulary and describes an ETL job — extract, transform, load — that moves rows between two systems and touches no schema at all. Cases adv-02 and adv-04 are the shape that catches skills whose descriptions name a subject rather than a task — the subject is identical and only the verb differs.

The fourth source is the disagreement pile from the labelling pass. Those nineteen cases where two labellers split are not a defect in the corpus; they are the requests where the skill's boundary is genuinely ambiguous, and they belong in the suite with whatever the adjudicator decided, flagged so that a failure on one of them is read differently from a failure on a clear case.

Two rates, and a gate that can fail in both directions#

With both halves present the harness reports a confusion matrix rather than a score — the four counts you get from crossing what the label expected against what the runtime did — and every edit moves four cells instead of one.

from dataclasses import dataclass

@dataclass
class Result:
    tp: int = 0   # should fire, fired
    fn: int = 0   # should fire, did not
    fp: int = 0   # should not fire, fired
    tn: int = 0   # should not fire, did not

def evaluate(cases, skill: str) -> Result:
    r = Result()
    for case in cases:
        fired = skill in agent.run(case["request"]).selected_skills
        expected = case["expect_skill"]
        if expected and fired:      r.tp += 1
        elif expected and not fired: r.fn += 1
        elif not expected and fired: r.fp += 1
        else:                        r.tn += 1
    return r

r = evaluate(load("corpus/db-migrations.jsonl"), "db-migrations")
activation      = r.tp / (r.tp + r.fn)
false_activation = r.fp / (r.fp + r.tn)
print(f"activation {activation:.1%}  false activation {false_activation:.1%}")

Run on the combined set, the pre-rewrite description scored 17.1% activation and 1.2% false activation; the rewrite scored 79.2% and 24.9%. Both of those are defensible operating points for some skill somewhere, and neither is obviously right for this one. The comparison is only possible because the second half of the suite exists.

A rate measured on a suite is not the experience users have, because the suite's mix of positives and negatives is chosen and production's is not. In the widened pull, 240 of 1,100 sessions were migration work, so roughly 22% of database-adjacent traffic is what the skill is for. Take a thousand such sessions: 218 positives, of which 79.2% activate, gives 173 correct loads; 782 negatives, of which 24.9% activate, gives 195 incorrect ones. More than half of everything the skill did was wrong, on a suite where the activation number looked healthy. This is the ordinary arithmetic of a low base rate — the share of arriving traffic that the skill is genuinely for, 22% here — and it is why a false activation rate that reads as tolerable in isolation has to be multiplied out against the traffic mix before anyone decides it is acceptable.

Negatives are also a different thing from quality regressions, and the two get merged because both feel like "the skill made things worse". A negative case asks whether the skill's text should have entered the context for this request, and it is answered by looking at the routing decision alone. Whether the resulting answer was any good is a separate question with a separate measurement, and a case that produced a fine answer despite an unwanted activation is still a false activation — the four-minute verify step ran either way.

The third row is the version that shipped. It kept the domain vocabulary and added a sentence naming what the skill is not for — read-path performance analysis, data movement between systems, and questions about existing schema — which cost 4.6 points of activation and removed roughly three quarters of the false activations. That trade was made deliberately, with both numbers on the table, which is the whole point of having the second half.

The gate that follows from this is a two-sided one. A change ships when activation does not fall and false activation does not rise, or when a fall in one is accepted explicitly with the trade written down. Returning to the colleague who wants to ship a description change on the strength of 96% to 99%: the number that decides it is the false activation rate on the negative half, because the positives-only movement is compatible with any amount of collateral broadening, and the negatives are the only place that broadening shows up as a cost.

Both of these numbers are still about routing#

Everything measured so far concerns one decision: whether the skill's text entered the context. A skill can score 95% activation and 2% false activation, and still be entirely decorative, because the agent read its body, produced something that ignored the central rule, and finished. Neither cell of the confusion matrix would move. The next chapter is about how to tell whether the instructions changed anything at all, and why the obvious way to check that — asking a model whether the agent followed the rules — reliably reports success on runs that broke them.