Merging overlapping tools

A colleague sends you six tool definitions from an internal helpdesk agent that has been live for a month, and asks which two should be merged.

[
 {"name": "search_docs",
  "description": "Search the documentation corpus for pages matching a query.",
  "input_schema": {"query": "string", "limit": "integer"}},

 {"name": "find_document",
  "description": "Find documents relevant to a user's question.",
  "input_schema": {"question": "string", "top_k": "integer"}},

 {"name": "get_document_by_id",
  "description": "Retrieve the full text of one document.",
  "input_schema": {"document_id": "string"}},

 {"name": "list_tickets",
  "description": "List tickets, optionally filtered by state and assignee.",
  "input_schema": {"state": "string|null", "assignee": "string|null"}},

 {"name": "create_ticket",
  "description": "Create a support ticket.",
  "input_schema": {"title": "string", "body": "string",
                   "priority": {"enum": ["P4","P3","P2"]}}},

 {"name": "escalate_ticket",
  "description": "Create a P1 incident and page the on-call engineer.",
  "input_schema": {"title": "string", "body": "string", "impact": "string"}}
]

The pair is search_docs and find_document. Read their descriptions as a router would: one takes a query and matches pages, the other takes a question and finds relevant documents. There is no user request that belongs to one of them and not the other, and no wording in either description that a request could be tested against. Both are full-text retrieval over a documentation collection, differing only in the vocabulary of whoever wrote them and in whether the size argument is called limit or top_k.

What overlap does to the traffic#

The consequence is not that the model picks the worse tool. It is that neither tool has a definable behaviour any more. Over 300 runs, the helpdesk agent's retrieval calls split like this by intent:

Read down the columns before reading the consequences. On the first phrasing the split is a little over half one way and a little under half the other; on the third it is nearer two thirds against one third, in the same direction. On the middle phrasing it tips the other way entirely, and the majority goes to the tool that lost both of the other rows. The inconsistency is the finding, not the near-even split on its own. A tool that took every how do I question and none of the where is the doc questions would be routing badly, but it would be routing: you could state what it was doing in a sentence and then change the sentence. Here there is no rule to state. The same phrasing lands in different tools on different requests, which is what it looks like when the choice is being decided by something other than the words the user typed.

Those columns are close to what you would get from a weighted coin. Three things follow from that, and they are all worse than a wrong choice would be. Cache hit rates halve, because the same user question arrives at two different retrieval paths with two different embeddings and two different result caches. Any change to relevance tuning has to be made twice or it applies to half the traffic. And the per-tool metrics from chapter 2 stop meaning anything: a 4.9% invalid rate on one and 29% on the other does not tell you that one tool is broken, it tells you that whichever tool has the fiddlier argument names is absorbing the errors from a routing decision that is being made at random.

get_document_by_id looks superficially like a third member of the same family and is not a merge candidate, because it takes an argument nothing else takes and answers a different question. A model does not choose it by weighing a description; it chooses it because it is holding a document id, which is a fact about the conversation rather than a judgement about wording. That is the general shape of a split that works: the argument routes the call, and the description only has to confirm it.

The merged replacement is one tool with one argument name and one set of semantics:

{"name": "search_docs",
 "description": "Search documentation. Accepts a natural-language question or "
                "keywords; returns ranked passages with the page they came from.",
 "input_schema": {
   "query": {"type": "string",
             "description": "The user's question, in their own words"},
   "limit": {"type": "integer", "default": 5, "maximum": 20}}}

A merge is only finished when the second tool is deleted. Keeping both and adding a mode argument that selects between the old behaviours moves the coin flip from tool selection to argument selection and changes nothing else, which is a rename rather than a merge.

Running the routing table again after the change is worth the fifteen minutes, because a merge can look successful in review and move nothing measurable. Three weeks after the deletion, the surviving search_docs was carrying all 275 retrieval calls per 300 runs that the pair had been splitting; its invalid-argument rate settled at 6.1% against a call-weighted 14.3% across the two tools before, since the errors that had been concentrated in top_k disappeared with the argument; and the retrieval cache hit rate went from 31% to 58% because one question now produces one embedding and one lookup instead of two. Median retrieval calls per run fell from 1.9 to 1.4, and the pattern behind that drop — call one tool, get a thin result, call the other one to check — had been showing up in nineteen runs in a hundred.

The pair somebody will propose merging#

Within a week of that review, someone will propose merging create_ticket and escalate_ticket into a single create_ticket(title, body, priority, page_oncall). The argument is tidy: it is the same resource, the same table, the same POST, and eight tools with a P1 special case is worse than seven tools with an argument.

{"name": "create_ticket",
 "input_schema": {"title": "string", "body": "string",
                  "priority": {"enum": ["P4","P3","P2","P1"]},
                  "page_oncall": {"type": "boolean", "default": false}}}

The thing that makes these two tools different is not visible in the resource they touch. It is what happens after the call returns — the blast radius of a wrong call, meaning how far its effects reach outside the agent, who they land on, and how much work it takes to undo them. Creating a P4 ticket is reversible in three seconds by closing it, costs nobody anything, and is authorised for every user of the agent. Paging on-call wakes a specific human being at 3am, cannot be undone once their phone has rung, triggers an incident channel, pauses a deploy freeze timer in some organisations, and in most companies is restricted to a subset of staff. Those are not two settings of one operation.

Once the flag exists, the guard against it is a sentence. The model reaches the paging behaviour from the ordinary creation path by setting a boolean, and the only thing standing between an emphatic user and a woken engineer is whatever the description says about when page_oncall should be true — which is a rule executed, probabilistically, by the same component that produced the coin flip in the table above. A team that shadow-ran the merged shape for two weeks recorded 11 pages against 4 that a human triager agreed with. The seven false pages all came from user messages containing the words urgent, down, or asap, none of which the model has any way to calibrate.

Those three words are the mechanism rather than a coincidence in the sample. Severity in an incident process is defined by impact on the system — how many users are affected, which service, whether a workaround exists — and none of that is in front of the model. What is in front of it is the user's affect, which is the only severity-shaped signal a chat message carries. Affect and impact correlate weakly and are not the same variable: a user writing asap about a slow dashboard types the same word as a user whose checkout is down. So a boolean guarded by a sentence is being set from the one input that cannot tell those two apart, and the sentence would have to make the model discount the only evidence it has.

Keeping the tools separate makes the guard a permission instead of a sentence:

TOOL_POLICY = {
    "search_docs":        {"scopes": ["docs:read"]},
    "get_document_by_id": {"scopes": ["docs:read"]},
    "list_tickets":       {"scopes": ["tickets:read"]},
    "create_ticket":      {"scopes": ["tickets:write"]},
    "escalate_ticket":    {"scopes": ["incident:page"],
                           "require_confirmation": True,
                           "rate_limit": "3/day",
                           "audit": "pagerduty_incidents"},
}

def visible_tools(user):
    return [name for name, p in TOOL_POLICY.items()
            if set(p["scopes"]) <= user.scopes]

With the tools separate, a user without incident:page never sees escalate_ticket in their tool list, so the paging behaviour is not merely discouraged, it is unreachable. With the tools merged, that same restriction has to be re-expressed as a conditional check on an argument value inside a handler, which is possible but is a check somebody has to remember to write, and which fails open by default. Separate tools also give you a rate limit, a confirmation step, and an audit trail that has a tool name in it rather than a field.

The word unreachable is doing precise work in that sentence, and unpacking it is worth a paragraph, because the difference between a permission and a description is mechanical rather than cultural. visible_tools runs on your server, before the request to the model is assembled. A tool it filters out has no name, no description and no schema anywhere in the token sequence the model receives, so there is no continuation the model could produce that names it — and a tool call for something absent from the tools array is refused by the API before your handler is reached. A description sits inside that same token sequence. It is one more piece of text competing with everything else in the context, including a user typing this is a total outage, page someone now. The guard and the thing it is guarding against are made of the same material, and the material is probabilistic. That is the whole of the argument, and none of it depends on how well the sentence is written.

Two axes, applied to the six#

Every merge decision comes down to two questions asked in order. Do these tools overlap in intent to the point where the model cannot route between them from the request alone? And do they differ in what happens if the call is wrong?

Rows one and two match on every column, which is what a merge candidate looks like. Rows five and six differ on the two columns that matter most, which is what a refusal looks like. The middle rows are neither: they overlap with nothing and they are cheap to get wrong.

The same reading applies to the customer-support surface from chapter 2, where the routing table was already showing a 47/37 split. search_kb and lookup_article merge, for exactly the reasons above; create_ticket and escalate_ticket do not; and get_article(id) and get_ticket(id) are separate despite both taking an id, because the ids come from different systems and a merged get(id) would have to guess which.

The corner where both answers are wrong#

The two-axis table has an empty corner, and surfaces land in it often enough that the rule needs the third branch the flowchart gives it. Two tools can overlap in intent completely and differ in blast radius, which is the combination that produces the longest design reviews, because each side of the argument is reading a different column and both are right.

The CRM half of the same helpdesk agent has the pair. link_contacts(a, b) records that two contact records are believed to be the same person and is undone by deleting the link. merge_contacts(primary, secondary) folds the secondary record's activity, notes and email history into the primary and deletes it, which the CRM has no undo for — support's runbook for a bad merge is a restore from the nightly export followed by a manual replay of anything written since. Both tools are the answer to these two are the same person, sort it out, and both descriptions say so.

Merging them fails the second question and leaving them as they stand fails the first, which the last row makes concrete: twenty-one of the thirty-three merges were calls a human would have made as links, and each one cost a restore. The resolution is neither branch of the original rule. Keep the two tools separate, then remove the routing ambiguity by requiring the irreversible one to carry evidence the reversible one does not — merge_contacts gains a required confirmed_by argument holding the id of a human approval record, which the model cannot manufacture and cannot obtain without asking someone. The distinction stops being a judgement about wording and becomes a fact about whether an approval exists, which is the same property that keeps get_document_by_id out of trouble.

That generalises past this pair. Where overlap and blast radius both apply, the argument separating the dangerous tool from the safe one has to be something the model cannot supply from its own reasoning: an approval id, a confirmation token, or a value returned by an earlier read that only exists when the precondition holds. A required argument the model can invent is a description with a colon in it, and it will be satisfied on every call that the model was going to make anyway.

Prompts change probabilities, schemas change reachability#

The predictable response is that tool count is aesthetics and the real fix is a better system prompt — add a paragraph explaining that search_docs is for keyword lookups and find_document is for natural-language questions, and the split resolves itself.

It does move, and not far enough to matter. The team above tried it: three sentences of routing guidance shifted the first row of the traffic table from 54/42 to 68/28, and left the invalid-argument rate where it was, because a request arriving as a natural-language question that the user phrased with keywords is genuinely ambiguous and no amount of prose resolves an ambiguity that is in the request rather than in the model. The deeper difference is structural. A prompt changes the probability of a call; a schema changes which calls exist. Deleting a tool takes the wrong branch out of the reachable set entirely, and it does so at zero marginal cost, whereas the three sentences of guidance are input tokens re-sent on every turn of every request for as long as the agent runs.

Put the two splits side by side and the size of that win is easy to misjudge. Before the guidance, a bit under half of the row went to the tool that should not have had it; afterwards, a bit under a third did. Roughly a third of the misrouting on that row came out, which is a real effect and is why the intervention keeps being proposed. It is also the wrong shape of improvement to build on. What it moves is a probability, so no quantity of further guidance takes the wrong branch to zero, and every attempt is paid for in tokens on every turn for as long as the agent runs. Deleting the tool takes the same errors to zero in one edit and costs nothing afterwards, because the branch it removes is not less likely — it is absent.

There is a case where none of this applies, and it is worth naming so the rule does not get over-applied. Two tools with overlapping intent are sometimes not a merge but a deletion: if a second retrieval tool was added six months after the first by a team that did not know it existed, and its retrieval path is strictly worse, the right change is to delete it and keep one. Merging implies you want both behaviours in one contract. Frequently you want one behaviour and a smaller surface.

Two tools that overlap in intent are a routing defect and a merge fixes it; two tools that differ in blast radius are a safety boundary and merging them dissolves it into a sentence. When both apply, the fix is a required argument the model cannot produce on its own. None of the three answers is about tool count, which is why counting tools is a poor way to review a surface.

Merging fixes routing. It does nothing to the inside of the surviving tool, and the surviving search_docs in that helpdesk agent still hands the model a cursor, twenty rows at a time, with an HTTP status code attached — which is the shape the next chapter takes apart argument by argument.