Generating tools from a spec

The proposal arrives as a working branch, which is the strongest form it can take. Someone has pointed a generator at openapi.yaml, and 140 tool definitions came out the other end in about forty minutes. The pitch has three clauses and each of them is true: it is complete, so no capability is missing; it is automatic, so nobody hand-writes schemas; and it stays in sync, so when the platform team adds an endpoint next quarter the agent gets it for free. Against that, a hand-built surface of twelve tools looks like a smaller product built more slowly.

Take the third clause seriously, because it is the one that decides the argument. Staying in sync with the internal API means the model-facing contract changes whenever the internal API changes, and internal APIs change for internal reasons. When the platform team splits /v3/invoices into /v4/invoices and /v4/credit-notes to fix a data-modelling problem, the agent's tool list changes underneath it, evaluation baselines from chapter 2 stop being comparable, and the prompt-cache prefix invalidates on the deploy. Synchronisation with something that moves for reasons unrelated to your task is coupling rather than a feature.

The steps underneath that sentence are worth laying out, because stays in sync is the clause everyone nods at. The generated tool list is a function of the specification. The specification is a function of the internal API. The internal API changes when a platform team has an internal reason to change it — a data-modelling problem, a deprecation, a performance fix — and none of those reasons has anything to do with what your agent is for. So a deploy nobody told you about rewrites the model's vocabulary, and three consequences follow at once. Evaluation baselines were measured against a tool list that no longer exists, so the comparison against last quarter is void. The prompt-cache prefix begins with the tool block, so it invalidates for every session in flight. And any description work done on the affected tools is gone, because the generator writes over it. A hand-built layer facing the same refactor changes in one place, on purpose, when somebody decides it should.

What the generator actually produces#

Here is one endpoint out of the 140 and the tool definition it becomes. The generator is doing a competent job; there is no bug in it.

# openapi.yaml
/v3/collections/{collection_id}/documents/search:
  get:
    operationId: searchCollectionDocuments
    summary: Search documents within a collection.
    parameters:
      - {name: collection_id, in: path, required: true, schema: {type: string, format: uuid}}
      - {name: q, in: query, schema: {type: string}}
      - {name: cursor, in: query, schema: {type: string}}
      - {name: page_size, in: query, schema: {type: integer, maximum: 20}}
    responses:
      "200": {$ref: "#/components/schemas/DocumentPage"}
      "404": {description: No matching documents}
      "409": {description: Collection is reindexing}

{"name": "searchCollectionDocuments",
 "description": "Search documents within a collection.",
 "input_schema": {"collection_id": "string (uuid)", "q": "string",
                  "cursor": "string", "page_size": "integer, max 20"}}

That is the tool chapter 6 spent a chapter dismantling, reproduced exactly, and the generator will reproduce it again on the next run. Every fix that chapter described — absorbing the paging, replacing the UUID with an area name, translating 404 and 409 into task-level outcomes — is a change the generator overwrites, which means a generated surface cannot be incrementally improved. It can only be replaced or annotated with an ever-growing override file that is a hand-built surface wearing a generator's clothes.

Multiply by 140 and three failures compound that a twelve-tool surface does not have.

The first is that every one of them is endpoint-shaped, so the plumbing tax from chapter 1 is charged on every workflow rather than on one. Nothing in the 140 corresponds to find this customer's most recent unpaid invoice and email it, because no endpoint corresponds to it; that workflow is a composition of seven, and composition is exactly what the model is bad at and what a purpose-built tool does in code.

The second is density. A REST API with 28 resources and five verbs each is not 140 unrelated operations; it is five families of 28, and within each family the descriptions are near-identical by construction. The similarity screen from chapter 7 found 190 confusable pairs in this document, and the worst of them were not the parallel list_* verbs — those route cleanly on the object name — but the twelve legacy endpoints kept for backwards compatibility, where searchCollectionDocuments, queryDocuments and findFiles all exist and two of them are deprecated in a way the spec records as a deprecated: true flag that carries no weight in a description.

Multiply those two counts and the catalogue size comes out exactly: twenty-eight objects, five things you can do to each, and the product is the hundred and forty. That factorisation is what makes the density predictable before anything ships rather than discoverable afterwards. A generated surface has one description per cell of a grid, and each description is assembled from its row label and its column label, so every cell shares a verb with twenty-seven others and shares an object with four. Confusability is not an accident of wording here. It is the structure of the specification being reproduced faithfully, which is what the generator was asked to do.

The third is blast radius. Generating from a spec inherits no authorization model, so all 140 tools run under whatever single credential the agent holds, and that set includes deleteWorkspace, patchBillingPlan and rotateOrgApiKey. The two-axis review from chapter 5 asks whether tools differ in what happens when the call is wrong; a generator does not ask, and cannot, because reversibility is not a field in OpenAPI. On this document, 9 of the 140 operations were irreversible and administrative, and every one of them was one selection error away from being called.

One operation in fifteen or so does not sound alarming until the denominator is read properly. The number that matters is not the share of the catalogue that is destructive; it is how often a destructive tool is among the candidates being ranked. Every turn of every run ranks all hundred and forty, so those nine sit in the comparison on every decision the agent makes, including the ones about searching documentation. The previous chapter measured what happens to selection at that catalogue size, and the answer was that six percent of calls come back invalid and more than a quarter of first calls land on the wrong tool. Those two measurements belong in the same sentence: a high misroute rate and an unguarded destructive operation are not two independent problems on one surface.

Filtering to the top ten does not rescue it#

The counter-proposal, offered within minutes of the first objection, is that the model never sees 140 tools. Embed the user's request, embed the 140 descriptions, send the ten most similar. The catalogue is 140 and the prompt carries 12; the accuracy problem from the previous chapter is solved by retrieval.

It fails in a specific and measurable way, and the measurement is recall of the required tool set, recall being the share of tasks for which every tool a correct run needs survived the filter. Label each task in the frozen set with the tools a correct run needs, run the filter on the opening user message, and count how often all of them are present in the window.

def recall_at_k(tasks, retrieve, k=10):
    full, partial = 0, 0
    for t in tasks:
        window = set(retrieve(t["request"], k=k))
        need = set(t["required_tools"])
        full += need <= window
        partial += bool(need & window)
    return full / len(tasks), partial / len(tasks)

# 200 tasks, k=10, embeddings over OpenAPI summaries
# all required tools present: 46%   at least one present: 91%

Ninety-one percent of tasks got something relevant and 46% got everything they needed, which is the number that decides it, because a workflow missing its fourth tool does not degrade gracefully. Single-tool tasks recalled at 78%; multi-tool workflows recalled at 31%, since the filter is computed on an opening sentence that describes the goal and not the intermediate steps. Email Acme their latest unpaid invoice is lexically about invoices and email, so the contact-lookup endpoint sits outside the window, and the run fails at step five.

The distance between those two figures is the first chapter's exponent arriving somewhere new. If a required tool has some fixed chance of surviving the filter, and a workflow needs several of them, then the chance that all of them survive is that per-tool chance multiplied by itself once per tool. Raising a bit over three quarters to the fourth power gives a little under four in ten, which is the neighbourhood the multi-tool figure sits in — and the measurement comes in below even that, because the chances are not independent in the direction that would help. The tools a workflow needs late are systematically the ones least similar to its opening sentence, so a workflow that has already lost one tool is more likely than average to have lost another.

The headline figure is a blend of those two, weighted by how many tasks in the set need only one tool. Working backwards from the blend and its two components puts roughly a third of the frozen set in the single-tool category, which is worth knowing before the number is quoted anywhere: change the mix of the task set and the forty-six percent moves without anything about the retrieval having changed at all. A task set written around simple lookups would make this architecture look considerably better than it is, and a set written around the workflows people actually run would make it look worse.

That failure is invisible to the instrumentation. The model does not emit an invalid call, because it never calls anything; it emits a sentence saying it cannot look up contacts, the run is recorded as an ordinary unsuccessful task, and the per-tool table shows nothing unusual because the missing tool has no row. Coverage failures produced by retrieval are the only failure mode in this topic that leaves no artefact behind, which is why teams running this configuration usually discover it from a support ticket rather than from a dashboard.

Two further problems arrive with the fix. Retrieval by similarity actively selects for near-neighbours, so the ten survivors are the densest cluster in the whole catalogue: a request about searching documents pulls searchCollectionDocuments, queryDocuments and findFiles into the same window together, which is the confusable set from chapter 5 assembled deliberately. And a tool block that changes per request is a prompt prefix that changes per request, so the caching that made the token argument survivable stops applying — 12 cached tools become 10 uncached ones, and the measured saving against a static twelve-tool surface was 8% of input tokens rather than the order of magnitude the proposal implies.

Two quantities get slid between in that proposal, and separating them is most of the answer. A hundred and forty tools in a catalogue against ten in a prompt looks like an order of magnitude, but the catalogue was never in the prompt — only the selected tools ever were. The saving actually on offer was the difference between ten definitions and twelve, which is small and points the wrong way about as often as not. Then the caching goes. A block that changes with every request is a prefix that matches nothing, so ten definitions bill at the full input rate where twelve had been billing at a fraction of it. Fewer tokens at full price is not automatically cheaper than more tokens at a tenth, and which way it comes out is arithmetic somebody has to do rather than a direction anybody can assume.

Retrieval over tools is a real technique with a real place, which is on top of a surface that is already well-shaped. Sixty curated, mutually exclusive, task-shaped tools with a load-on-demand mechanism is a sound architecture. The technique cannot repair operations that are the wrong shape, because it only ever changes which of them the model can see.

Building from traffic instead of from the specification#

Start from what people ask for. Pull 30 days of real requests — support tickets, Slack messages, whatever the agent is replacing — cluster them by intent, and read the distribution. On the billing agent that runs through this topic, 2,900 requests collapsed into 14 intents covering 84% of volume, with a tail of several hundred one-off asks. Those fourteen intents became twelve tools, each built over the existing API in the shape chapter 3 described, absorbing its own mechanics as chapter 6 laid out, and carrying its own scope as chapter 5 required.

That eighty-four percent is the whole justification for a twelve-tool surface, and it should be read as a decision rather than as a discovery. Sixteen percent of twenty-nine hundred requests is somewhere over four hundred asks a month the surface will not serve, and building for the head is choosing to route those four hundred somewhere else on purpose. The alternatives are worse in ways the number makes visible: a surface built for total coverage is the hundred and forty, and a surface built for the top three intents leaves a tail large enough that nobody would deploy the agent. Fourteen intents is where the curve flattens, and the curve is only visible because somebody clustered a month of real requests rather than reading a specification.

The bottom two rows are where a designed surface earns its keep, and the first of them needs a piece of construction that is easy to leave out. Give the agent an explicit tool for the tail.

{"name": "record_unsupported_request",
 "description": "Use when the user's request cannot be completed with the "
                "available tools. Records the request for a human and tells "
                "the user it has been passed on.",
 "input_schema": {
   "user_goal": {"type": "string", "description": "What they were trying to do"},
   "blocked_by": {"type": "string",
                  "description": "The step you could not perform"}}}

That tool converts the 16% into a queue with a row per miss, and the queue is the roadmap: after six weeks it said that 41% of misses were credit-note enquiries, which became tool thirteen, and that 22% were people asking for things the company does not permit over chat, which became a policy sentence rather than a tool. A generated surface has no equivalent, because its coverage is nominally total and its misses look like ordinary failures.

Both of those shares are shares of the tail rather than of the traffic, and converting them is what turns a queue into a roadmap. Two fifths of a sixteen percent tail is a bit over six percent of everything, which is the coverage the thirteenth tool buys — comparable to what several of the original twelve are worth, and considerably more than one more endpoint off a specification would have been. The other fifth of the tail buys no coverage at all, because the correct response to it is a policy sentence and not a tool. Getting either finding requires the misses to be grouped by what the user was trying to do, which is a thing only an explicit tool can record, because the model is the only component in the system that knows a request was refused for want of a capability.

The honest costs of the designed surface#

Twelve tools do not cover everything, and the 16% is real traffic that a human now handles. The surface is hand-maintained and will lag the platform team by a sprint or two whenever a new capability lands. Somebody has to own the unsupported queue or it becomes a log nobody reads. And the whole thing needs a review each quarter against the current traffic clusters rather than the ones from launch. Those are the price of the twenty-point difference in the table, and a proposal that does not name them will be read as advocacy.

Proving the call takes the four-arm discipline from chapter 2 with the arms named in that table, run against one frozen task set that deliberately includes tail tasks the twelve-tool surface cannot serve — leaving them out inflates the designed surface's number and makes the comparison worthless. Report task success, first-call correct, invalid rate, tokens per turn, and the unsupported-queue rate side by side, then shadow the winner against production traffic for two weeks with the tools executing in dry-run mode, comparing what the agent would have called against what the human did.

A generator answers the question "what can this API do", and a tool surface has to answer "what does this agent do". Those are different questions, and no amount of filtering, retrieval or deferred loading converts an answer to the first into an answer to the second, because every mechanism of that kind changes which operations the model can see rather than what shape they are.

What all of this leaves you is a procedure rather than a preference. Instrument before you change anything; decide whether the existing API is already task-shaped or whether the task needs its own contract; choose granularity on the six criteria rather than on tool count; merge on overlapping intent and refuse to merge across blast radius; absorb transport inside the tool; and measure the catalogue as a whole, because its costs hide from every per-tool metric you have. The surface you get from that is small enough to read in one sitting, which is the property that makes everything downstream — evaluation, permissions, incident review, the next tool somebody asks for — tractable.