Past that number the library needs routing, not fewer capabilities

The platform team took the 90% accuracy floor to their director with a recommendation to cap the catalogue at 130 skills. The meeting lasted eleven minutes. There are 402 skills in the registry; one of them encodes the escalation policy for the payments on-call rota and is used about eleven times a week by a team of nine people who wrote it themselves after a bad night; another walks through the six-step process for issuing a partial refund across two ledgers, and it is the only written record of that process anywhere in the company. A cap of 130 is a decision to delete 272 things that work and have users, and no director signs that. The requirement is the catalogue. What has to give is the assumption that every request weighs every skill.

That assumption is worth stating explicitly, because it is the only place the arithmetic can be attacked. Resting cost is not a property of how many skills exist; it is the number of descriptions resident in the prompt of a given request, multiplied by their average size. A registry of 402 skills and a request that sees 40 of them costs the same as a registry of 40. Every architecture below is a different answer to the question of which subset a given request sees, and each one buys back resident tokens by introducing a decision that can be wrong in a new way.

Four ways to stop showing every request the whole catalogue#

The cheapest change structurally is to scope visibility by surface. The 402 skills are not equally plausible for every agent the company runs; the IDE assistant has no business selecting the refund-issuance skill, and the customer-support agent has no business selecting the one that rewrites Terraform modules. Tagging each skill with the surfaces it belongs to and assembling the resident block per surface is a configuration change rather than a system:

# registry/surfaces.yaml
surfaces:
  ide:
    include_tags: [code, build, test, deploy-read]
    exclude_tags: [customer-data, finance]
  support-console:
    include_tags: [customer-data, billing, webhooks]
    max_resident_tokens: 2000
  ops-oncall:
    include_tags: [incident, deploy-write, oncall, observability]

Across the platform team's six surfaces this put an average of 58 skills in front of any given request, for about 1,390 resident tokens, and selection accuracy on the labelled set came back to 92%. The new failure is the request that legitimately wants something outside its surface. An engineer in the IDE asks the assistant to page the on-call for the service they have just broken, and the escalation skill is not in the resident block, so the model cannot select it and answers from general knowledge about PagerDuty. On their traffic that was 4% of IDE requests, and none of it appeared in any log as an error, because a skill that is not visible produces no evidence that it was wanted.

The second option is to retrieve the descriptions instead of resident-loading them: embed the user's request, run it against an index of the 402 descriptions, and put the top fifteen into the prompt.

def resident_block(request: str, k: int = 15) -> str:
    q = embed(request)                       # ~35 ms, 1 API call
    hits = index.search(q, k=k)              # cosine over 402 vectors
    return "\n".join(
        f"{h.name}: {h.description}" for h in hits
    )

Fifteen descriptions is about 360 resident tokens, the smallest number any of these options reaches, and it scales to 4,000 skills as easily as to 400. What it reintroduces is a retrieval system with a recall ceiling — recall being the fraction of requests whose correct skill appears anywhere in the returned set, which bounds what the model can pick from however well it picks. On the platform team's labelled set, the correct skill appeared in the top fifteen for 93% of requests, which caps end-to-end selection accuracy at 93% no matter how good the model's final choice is. Recall failures are also not randomly distributed: embeddings miss when the request's vocabulary and the description's vocabulary diverge, which is the same population of requests that selection was already worst on. There is a second cost that is easy to miss. The resident block is now different on every request, so the prompt prefix is no longer stable and the cache hit that made 20,000 tokens affordable in the first place is gone for everything downstream of that block.

Third is a coarse router: group the 402 skills into twelve domains, make the domain the thing that is resident, and let the model choose twice.

The model never discriminates among 402 candidates; it discriminates among 12 and then among 33, and both of those are inside the range where accuracy was measured at 91% or better. What it costs is an extra round trip, about 620 ms on the platform team's p50, and a compounding error rate: 97% correct on the domain step and 94% within the domain gives 91% overall, which is worse than scoping achieved. A wrong domain is also unrecoverable unless the runtime lets the model back out and try another one, and every backtrack is another round trip.

The fourth option does not change the architecture at all. Three of the four near-neighbour clusters — the log-query cluster, the report-generation cluster, the environment-reset cluster — exist because separate groups solved the same problem separately. Merging each cluster into one skill whose body branches on the case removes the ambiguity rather than routing around it. Merging took the registry from 402 to 310 and lifted accuracy on requests belonging to those clusters from 61% to 88%. The cost lands on the body: a merged skill's body has to cover three cases, so it is roughly 2,200 tokens instead of 900, and every run that selects it pays for all three. Ownership becomes the harder problem, because a skill that three teams depend on and one team owns is a skill that gets stale in the two directions its owner does not use.

What each one actually buys#

The column that decides this is rarely the token column, which is the point the arithmetic keeps hiding. All four options land between 360 and 7,440 resident tokens, and any of those is affordable against a large context window. They differ in what they break and in who has to keep them working, and those differences are a factor of ten apart while the token numbers are within a factor of twenty of each other but all comfortably inside budget.

One detail in that table is an average pretending to be a budget. Scoping's 1,390 resident tokens is the mean across six surfaces, and means are the wrong statistic for a cost you pay per request. Their ops-oncall surface carried 141 skills, or 3,380 tokens, because incident work touches nearly everything; the IDE surface carried 22. Whichever surface has the tightest latency budget is the one whose number matters, and on their deployment that was the IDE at 22 skills, which is fine, and could just as easily have been the reverse. The resting cost of a scoped architecture is therefore a vector with one entry per surface, each computed the same way as the single number from the previous chapter, and the one to write into the design doc is the largest entry on the surface with the smallest budget rather than the arithmetic mean across all of them.

The platform team shipped scoping and merging together and skipped the other two. Merging first, because the clusters were a description problem that no routing layer could fix — retrieval over three descriptions that all match equally returns all three, and a router sends them all to the same domain. Then scoping, because it added no latency and no new subsystem, and because their traffic was genuinely partitioned by surface. They kept the 90% floor as the gate and added a second metric: the rate of requests where the model selected nothing on a surface where a scoped-out skill would have matched, sampled by running 200 requests a week against the unscoped catalogue in shadow — replaying real traffic through a second configuration whose answers are recorded and never returned to anyone — and comparing. That shadow run is what turns the invisible failure of scoping into a number.

The problems routing does not touch#

Routing changes which descriptions compete. It does nothing about descriptions that do not describe the situations that should trigger them, and a badly written description fails identically at 402 candidates and at 15. Half of the platform team's misroutes, when they read the transcripts, were requests where the correct skill was resident, visible and unambiguous, and the description simply did not mention the phrase the user had used. No amount of narrowing the candidate set repairs that.

It also does nothing about the second and third tiers. Resident cost is a routing-layer problem; the tokens a run spends after selection — the body, the reference files — are unaffected by whether the model chose from 12 candidates or 402. A team that fixes resting cost and then loads a 6,000-token body on every run has moved the expense rather than removed it.

And there is a class of request none of these handle well: the one that needs three skills at once. Migrate this service, which means reading the deploy skill, the database-migration skill and the rollback skill. Scoping is neutral on it, merging cannot help unless the three belong together, and both retrieval and the two-step router are biased against it because both are built to return the best match. Multi-skill requests were 6% of the platform team's traffic and had the worst outcomes of any category under every architecture they tried, and that remains unsolved for them.

What all four options share is a dependence on the description carrying enough signal to route on while staying small enough to be resident on every request. Scoping decides which descriptions are in the room; it does not make them better. That puts the weight back on a question this chapter has assumed an answer to: given a skill with prerequisites, a procedure, three worked examples and a 900-token schema, which of those sentences belong in the description that is always resident, which belong in the body, and which belong in a file nobody reads unless the body asks.