Near-neighbour density in a tool catalogue

An eval, throughout this topic, is a labelled set of requests replayed against the agent and scored: for each request, a human has written down which tool was correct, and the score is the share the model got right. It is the offline test that stands in for production behaviour, and it is the only instrument that can say anything about selection at all, because chapters 1 through 3 established that no schema, validator or value set constrains which tool gets chosen.

What an eval reports by default is one number, and one number is exactly enough information to reach the wrong conclusion about why a catalogue got worse. This chapter is about the structure underneath that number — which pairs of tools the failures fall into — and about the control run that distinguishes the two explanations available for any drop, because those two explanations imply completely different work and only one of them is usually true.

Which pairs the failures fell into#

Selection errors are not distributed across a catalogue. On one support agent that went from twelve tools scoring 96% on a labelled set of 400 requests — 384 of them routed to the tool a human had marked as correct — to thirty tools scoring 71%, 94 of the 116 failures came from seven pairs of tools:

Both counts in that sentence are worth reconstructing, because a table of pairs is only evidence if it accounts for the whole loss rather than a chosen part of it. The labelled set holds four hundred requests, and seventy-one percent of four hundred is two hundred and eighty-four routed correctly, which leaves the hundred and sixteen that did not. Adding the Failures column gives ninety-four, and ninety-four out of a hundred and sixteen is a little over four fifths. That is the figure the argument rests on, and it survives only because the column was summed rather than eyeballed — seven rows chosen for being large will always look impressive, and the question is what fraction of the total they are.

The remaining 22 failures were scattered singletons across the other 23 tools. Every one of the seven heavy pairs has the same shape: an original tool and one of the eighteen new ones, where the new tool is a plausible answer to the same class of request. A customer says "I want my money back", and refund_order, issue_credit_note, void_charge and apply_store_credit are four operations that all return money to a customer, differ by settlement mechanism and accounting treatment, and are described in the catalogue in terms of that mechanism. The distinction between them is a finance distinction that the conversation does not contain.

The count explanation predicts something different. If thirty tools is past a threshold where the model's ability to attend to a long tool block degrades, failures should spread — long-tail tools should suffer, tools described late in the block should suffer more than early ones, and the pairs should look arbitrary. What actually happened is that 81% of the loss lands on seven pairs out of the 435 possible pairs in a thirty-tool catalogue.

The denominator in that comparison is the part that makes it damning, and it is worth deriving rather than accepting. Every tool can in principle be confused with every tool that is not itself, which is twenty-nine partners for each of thirty tools. Multiplying gives eight hundred and seventy, and each pair has been counted from both ends, so halving it gives the four hundred and thirty-five. Seven of those carry four fifths of the loss, which is under two in a hundred of the available pairs carrying almost all of the damage. Spread would look nothing like that. If attention across a longer tool block were the mechanism, the failures would be distributed across a large share of the four hundred and thirty-five, because nothing about running out of attention picks out particular partners.

Setting the two explanations beside each other makes them testable rather than rhetorical, since they disagree about several things the same eval run already recorded.

Three of those five rows can be read off the run that produced the 71% without doing anything new. The failures were concentrated rather than spread. Position was not predictive: the eighteen new tools were appended after the original twelve, and the twelve originals absorbed roughly as many wrong choices as they generated, which is the opposite of what a serialisation-order effect looks like. Shuffling the block across three orderings moved the aggregate by 1.4 points and left the seven pairs exactly where they were. The third row is the one that needs a run of its own, and it is cheap.

The control that settles it#

The cheap way to break the tie is to add eighteen different tools instead. Take the same twelve-tool agent, add eighteen tools from the merchandising and content domain — product detail lookups, size charts, store locations, warranty terms, campaign metadata — none of which is a plausible answer to a support request about an order, and rerun the same 400 labelled requests.

Same model, same prompt, same 400 requests, same count of thirty in rows two and three. The count is held constant and the accuracy differs by 22 points, so the count is not what is doing the work. The 2.8 points that the merchandising set does cost is real and mostly attributable to one pair — get_return_policy against get_warranty_terms, which are genuinely confusable — plus a small uniform tax from a longer tool block. The fourth row is the useful one for anybody arguing that count must matter eventually: forty-eight tools cost only 1.5 points more than thirty, because the second eighteen added almost no new near neighbours.

The Loss column carries no information the accuracy column does not — each entry is the baseline minus that row — and its job is to make the interesting comparisons subtractions between rows rather than mental arithmetic. Read that way, the last row says something the accuracy figures obscure. Going from thirty tools to forty-eight costs the difference between the two losses, which is the one and a half points, and that increment covers eighteen more tools than the row above it. The same eighteen tools cost twenty-five points when they were near neighbours of what was already there and one and a half when they were not, which is the same eighteen tools, the same model and the same requests differing only in what they sit next to.

That third row is what makes the run a control rather than another data point, and the design rule generalises past this table. Two explanations were on offer and they agreed about everything except one variable, so the run was built to hold everything else fixed and move that one: same model, same prompt, same labelled set, same catalogue size, different neighbours. A row where both explanations predict the same outcome is worth nothing however it comes back, which is why adding more tools of the same kind — the obvious next experiment — would have produced a number that neither side could have used.

Measuring overlap before you ship the tool#

You can get an advance warning of which additions will hurt without running an eval at all, by embedding the descriptions and looking at what each new tool is closest to. Embedding a piece of text means running it through a model that returns a vector of a few hundred numbers positioned so that texts meaning similar things land near one another, which lets you score any two descriptions for similarity on a scale where 1.0 is the same direction and 0 is unrelated. This is a rough instrument and it is cheap enough to run in continuous integration (CI) on every pull request that touches a tool definition.

import numpy as np
from itertools import combinations

def describe(tool):
    """The text the model actually sees, not just the description field."""
    params = ", ".join(tool["input_schema"]["properties"])
    return f"{tool['name']}: {tool['description']} (parameters: {params})"

def confusable_pairs(tools, embed, threshold=0.82):
    vecs = embed([describe(t) for t in tools])
    vecs = vecs / np.linalg.norm(vecs, axis=1, keepdims=True)
    out = []
    for (i, a), (j, b) in combinations(enumerate(tools), 2):
        sim = float(vecs[i] @ vecs[j])
        if sim >= threshold:
            out.append((a["name"], b["name"], round(sim, 3)))
    return sorted(out, key=lambda r: -r[2])

for a, b, sim in confusable_pairs(CATALOGUE, embed_texts):
    print(f"{sim:.3f}  {a} <-> {b}")

On the thirty-tool catalogue this prints refund_order <-> issue_credit_note at 0.91, search_orders <-> search_customers at 0.88, cancel_order <-> cancel_subscription at 0.87, and six others above 0.82. Six of the seven pairs from the failure table appear in the top eight rows of the similarity list, which is a good enough correlation to be worth a CI check and nowhere near good enough to replace the eval.

The one it misses is instructive. get_return_status <-> get_refund sits at 0.74, below the threshold, because the descriptions talk about different objects — a return is a physical item coming back, a refund is money going out — and are lexically and semantically distinct. They are confusable anyway, because both are correct-sounding answers to "where's my money", and the thing that makes two tools near neighbours is overlap in which requests they are a plausible answer to, not overlap in what they do. Description similarity is a proxy for that and an imperfect one. It will also generate false alarms in the other direction: get_order and cancel_order embed close together and are almost never confused, because the request always contains an unambiguous verb.

The model's own second choice is a better proxy

Description similarity guesses at confusability from outside the model, and the decision is happening inside it. On providers that return log probabilities for the sampled tokens — the model's own scores for each candidate it weighed at that position, before one of them was picked — you can read the margin directly instead of inferring it: run the labelled set, capture the top two candidate tool names per request, and record the gap between them.

resp = client.messages.create(
    model=MODEL_ID,
    messages=req.messages,
    tools=CATALOGUE,
    temperature=0,
    top_logprobs=5,
)
first, second = top_two_tool_candidates(resp)      # [(name, prob), (name, prob)]
rows.append({
    "request_id": req.id,
    "expected": req.label,
    "chosen": first[0],
    "runner_up": second[0],
    "margin": round(first[1] - second[1], 3),
})

On the twelve-tool catalogue the median margin was 0.81 and 4% of requests came in under 0.2. On the thirty-tool catalogue the median fell to 0.44 and 29% were under 0.2, and 88 of the 116 failures sit inside that low-margin group. The runner_up column is the half that pays for the instrument: on refund_order requests the second choice was issue_credit_note 61% of the time, which names the pair before anyone has labelled a single failure by hand.

The limit is that a confidently wrong answer has a large margin and looks healthy, so this measures how close the decision was rather than whether it was right. That makes it a screen for pairs worth investigating and a leading indicator for new tools: a pull request that halves the median margin on an existing tool's requests deserves review whatever it does to the aggregate, because it has landed a tool on top of one that was working. Providers that do not expose log probabilities for tool selection leave you with the embedding screen and the eval, which is a reason the eval remains the thing that decides.

Where the twenty-tool number came from, and what it is good for#

The rule of thumb has real ancestry. Function-calling benchmarks published from 2023 onward reported accuracy curves against catalogue size, several widely circulated vendor posts and conference talks put the knee somewhere in the teens to low twenties, and the MCP ecosystem gave everyone a way to attach four servers to one agent and land at sixty tools by accident, which made the failure mode common enough to acquire folklore. More recent work on large catalogues, including Dynamic ReAct: Scalable Tool Selection for Large-Scale MCP Environments, measures the collapse at genuinely large scale and tests retrieval as the mitigation; the curves there are steep, and they are curves for particular catalogues assembled from particular servers.

The number is a reasonable prior and a bad measurement. As a prior it says: if you are about to go from twelve tools to sixty, expect trouble and plan to measure. As a measurement of your system it asserts that your catalogue has the same density of near neighbours as whichever one produced the number, which is a claim nobody has checked. A thirty-tool catalogue of orthogonal operations can outperform an eighteen-tool catalogue containing four ways to move money, and the tables above are one instance of exactly that.

Two channels keep count relevant, and dismissing it entirely goes too far. The first is mechanical: the more tools you add, the higher the chance that any given addition lands near something already there, so density and count are correlated in practice even though density is the cause. A team adding tools at random will see accuracy fall roughly with count, and will reasonably conclude that count is the variable. The second is the tool block itself — thirty tools with full schemas is on the order of 6,000 to 9,000 tokens in front of every request, which costs money, eats context, and at large enough sizes does dilute attention independently of any confusion between pairs. The 1.5 points between thirty and forty-eight tools in the table is that effect, small at this scale and not zero.

The practical reading of a drop from 96% to 71% is therefore that the added tools are near neighbours of existing ones and the loss tracks the overlap rather than the count, and that reading points at different work than the count reading does. It says look at the seven pairs, ask whether refund_order and issue_credit_note should be one tool with a settlement parameter, ask whether search_orders and search_customers should be a single search with an entity type, and rewrite the descriptions of the pairs you keep in terms of when to use each rather than what each does.

None of the analysis above was available from the number 71%. The aggregate said the agent got worse; the pair table said which two tools to look at, and it is the pair table that turns a bad quarter into an afternoon's work. Yet the aggregate is what most eval harnesses print, what gets pasted into a retrospective, and what a teammate will quote when the next twelve tools are proposed.

That gap matters more than it sounds, because the aggregate does not merely lack detail — it can be stable while the thing you care about collapses. A catalogue that goes from 93% to 89% has lost four points, and four points is equally consistent with every tool losing four points and with two tools falling from 95% to 45% while the other twenty-eight hold exactly where they were. Those are completely different products, and the users of the two collapsed capabilities have gone from a working feature to a coin flip.