Tool-selection accuracy is meaningless without the no-tool cases

Tool-selection accuracy is the share of an agent's decisions on which it called the tool a human labeller had marked as correct. On a scheduling agent, the tool-selection report said 94.2% across 240 labelled decisions, which nobody argued with, and then somebody pasted a trace into the channel. A user had replied to the agent's proposal with "thanks, that works for me". The agent called list_availability for both attendees, again, then called read_thread, then produced its confirmation. Three tool calls, about 4,100 tokens and 2.3 seconds, to handle a message whose correct response was a two-line reply and a create_event that had already happened.

That decision was not in the labelled set, and no decision like it was. Every one of the 240 rows had been built the same way: take a turn from a production trace, look at what the agent did, write down the tool a competent engineer would have called. The instruction "write down the tool" had done all the damage on its own, because it presupposes that a tool is called.

Two policies the metric cannot tell apart#

Follow that presupposition through to the score. A row's label is a single tool name, scoring compares the name the agent emitted against it, and a row exists only where some call was the right one. Now hold two policies side by side. Both call the correct tool when a tool is needed; the first stops when nothing is needed, and the second reaches for something plausible. Every row in the set is a row where a tool was needed, so on every row the two emit the same name and score the same. The rows that would separate them are precisely the rows the construction excluded, which is why a metric built from those labels cannot distinguish an agent that chooses well from an agent that reaches for something on every turn: both score identically on every row in the set.

That is the shape worth recognising, because it is what makes a bad metric feel like a defence. The published number is pinned while the property it is trusted to govern, how often the agent acts when it should not, varies without limit. Believing that a high score protects you from an over-calling agent is tempting for three true reasons — the rows come from real production traces, the labels were written by engineers who know the system, and the number is high — and none of them touches the defect. What a metric can see is decided by which states enter its denominator, and care labelling a state cannot add a state that was never sampled.

The label is a set of acceptable actions, and one of them is nothing#

Tool selection is a classification problem, and it goes wrong in the ways classification problems go wrong when the label schema is built from the positive class alone — here, the turns on which something was called. The action space, meaning the set of things the agent may do at one decision point, has to include the actions that are not tool calls, and it has to admit that several actions can be right.

{
  "id": "tsel-0142",
  "state": {
    "thread_id": "th-8831",
    "last_message": "thanks, that works for me",
    "prior_actions": ["read_thread", "list_availability", "calendar.create_event", "mail.reply"]
  },
  "valid_actions": [
    {"tool": null, "reason": "confirmation received; nothing left to do"},
    {"tool": "mail.reply", "args": {"thread_id": "th-8831", "kind": "ack"},
     "reason": "a short acknowledgement is acceptable but not required"}
  ],
  "forbidden_actions": ["list_availability", "calendar.create_event"],
  "labelled_by": ["dh", "mk"]
}

Three fields there do work that a single expected_tool string cannot. valid_actions carries a set, because the alternative forces labellers to arbitrate between two genuinely acceptable choices, and that arbitration is where inter-labeller agreement collapses — on the first pass at this set, the two labellers disagreed on 38 of 240 rows, and 29 of those disagreements were both people picking a defensible action. tool: null is a first-class member of that set. forbidden_actions is separate from "not in the valid set", because there is a difference between an action that is merely unnecessary and one that is actively wrong, and mixing them produces a metric that treats a redundant read the same as an unwanted write.

Rebuilding the set with an explicit instruction to include turns where the right action is to stop changed its composition considerably. Of 240 rows, 31 came back with null in the valid set — confirmations, out-of-scope requests, messages that needed a clarifying question rather than a lookup, and threads where a required attendee had already declined and there was nothing to schedule.

Getting those 31 rows into the set required changing where the states came from. Sampling turns from traces and labelling what the agent should have done produces a set biased toward turns where the agent did something, because those are the turns that generate interesting-looking spans. Sampling states instead — every point at which the loop asked the model for a decision, including the final iteration where it decided to stop — gives a population that contains stopping decisions in the proportion they actually occur. Stratifying that sample by position in the run — drawing a fixed quota of states from each position bucket instead of sampling the whole population uniformly — helps further, since the no-tool decisions cluster at the end and a uniform sample of 240 decisions from runs averaging six steps will contain few of them by accident.

Those two moves are different in kind. Sampling states rather than turns is forced: a population with no stopping decisions in it cannot yield a row containing one, whatever the labelling instructions say. Stratifying by position is a choice somebody made and could have made differently. It buys enough rows in the rare class to compute a rate on, and it pays by making the class proportions a property of the quota rather than of production, which limits what the aggregate below is allowed to mean.

The single number was hiding a class it never contained#

With negatives in the set, the report splits and the aggregate stops being the interesting line.

The bottom two rows collapse the top three in different ways. Write nᵢ for the number of rows in class i, cᵢ for how many of those the agent got right, and rᵢ = cᵢ / nᵢ for that class's rate. The micro-average is total correct over total rows, (Σ cᵢ) / (Σ nᵢ) with both sums running over the classes, and it ignores the class boundaries entirely: its numerator is 179 + 21 + 13, which is 213, its denominator is 187 + 22 + 31, which is 240, and the division rounds to the 88.8% in the table. The macro-average is the unweighted mean of the per-class rates, (Σ rᵢ) / k for k classes, and it discards the row counts: adding 95.7%, 95.5% and 41.9% and dividing by three gives 77.7%. The two figures sit about eleven points apart, and that spread is the weight one gives to class size and the other does not.

Reading the finding off those two rows settles only this table; the symbolic version settles every table. Let wᵢ = nᵢ / N be class i's share of the N rows in total, so the shares are non-negative and sum to one. Substituting cᵢ = nᵢ rᵢ, which is the definition of rᵢ rearranged, turns the micro-average into (Σ nᵢ rᵢ) / N. Dividing a sum term by term is allowed because N is the same in every term, so that is Σ (nᵢ / N) rᵢ, and nᵢ / N is wᵢ by definition. The micro-average is therefore Σ wᵢ rᵢ — the weighted mean of exactly the rates the macro-average takes the plain mean of, each class weighted by its share of the rows.

The sensitivity of the aggregate falls out of that form. Hold every class but one fixed and move class j's rate by some amount d. The micro-average changes by wⱼ d, because rⱼ appears in the sum multiplied by wⱼ and nowhere else. The largest d available is one, a class going from every row wrong to every row right, so the most any class can move the micro-average is wⱼ, its own share of the rows. A class holding about an eighth of the denominator can go from near-total failure to perfection and shift the headline by about an eighth, which is how an aggregate stays high while one class collapses underneath it.

The original 94.2% is the limiting case, and the limit says something stronger than "the class was hard to see". That class was not merely small: its share of the rows was zero, because the construction that produced the rows could not produce one of them. Put wⱼ = 0 into wⱼ d and the product is zero for every d, so no behaviour on that class, however bad, could have moved the published figure at all. An insensitive number moves a little; this one was independent of the thing it was trusted to report.

The 41.9% is the whole finding, and it was invisible at 94.2% because the class was absent. As an analogy, this is the arithmetic that makes a fraud classifier look excellent while catching no fraud: when one class is 13% of the rows and the model never predicts it, the aggregate barely moves, which is wⱼ d with a small wⱼ. The analogy breaks on where the rarity comes from. A fraud classifier's positive class is rare in the world, so a small weight is a fact about the problem. The no-tool class was made rare by how the states were sampled, which is a defect in the instrument rather than a property of the task, so the first fix here is a different sample and only the second is a different average. Reporting the macro-average alongside the micro-average, which weights every row equally and therefore lets the largest class set the headline figure, is the standard defence, and it works here for the same reason it works there, by giving the small class the same weight as the large one in at least one published number.

Two of this chapter's own figures invite a comparison that is not available. The first report said 94.2% and the rebuilt one says 88.8%, which side by side looks like a fall of several points, and it is not a regression of any kind. They are two label sets over two populations of states, the second containing a class the first did not, so the same rows are not in both denominators and the two figures are not one quantity measured twice. Nothing about the agent changed between them. The number went down because the instrument started measuring something it previously could not see, and a comparison against the wrong reference is the error this chapter is about, arriving in its own results.

One structural fact licenses both averages. The three class rows partition the labelled set: every one of the 240 rows falls into exactly one of them, since a state either required a specific tool, or admitted several, or required none. Nothing is missing and nothing is double-counted, which is why 187 + 22 + 31 comes to 240, why the macro-average is the mean of exactly three rates, and why Σ wᵢ is one, so that Σ wᵢ rᵢ is a genuine weighted mean. What the partition does not license is any statement about production. The set it partitions is the labelled one, and its proportions — the 187, the 22, the 31 — were fixed by the decision to sample states rather than turns and by the per-position quota, both taken before a row was labelled. Change the quota and every wᵢ changes, so the micro-average moves without the agent moving.

The eighteen failures in the no-tool class were not uniform, and the breakdown is what turned the metric into a change. Eleven were redundant reads — read_thread or list_availability on a state where nothing new had arrived — which cost tokens and latency and nothing else. Four were clarifying questions the agent should have asked and instead answered by guessing, which is a correctness problem wearing a tool-selection costume. Three were forbidden actions: the agent called calendar.create_event on a thread where the meeting already existed, and produced exactly the duplicate that the fault-injection work in the last chapter — chapter 3 of this topic, which scores recovery by deliberately breaking a tool the agent depends on — was chasing from a different direction.

Eleven, four and three account for all eighteen, with no residue that resisted classification, and that exhaustiveness is what makes the breakdown a diagnosis rather than three anecdotes: a claim about the class as a whole can be checked against the three parts, because there is no fourth part for a counterexample to hide in. What the counts do not license is ranking the parts by importance, since the largest group is the cheapest per occurrence.

The metric is silent about arguments#

calendar.create_event is the right tool for a turn where the user has agreed to a slot. Called with start=2025-03-19T14:00:00Z when the user meant 14:00 in Europe/Madrid, it books a meeting an hour off and scores as a hit, because the tool name matched. Tool-selection accuracy as normally computed is a metric over names.

Extending it to arguments requires deciding what counts as a match, and the options differ sharply in cost and in what they let through.

The table teaches through the differences between rows rather than the levels, because each row is worth what it catches that the row above lets through. Exact equality flags every difference including the ones that are not defects, so its characteristic failure is a false alarm on "Q3 review" against "Q3 Review". Normalising per field type removes that whole class of false alarm by mapping both spellings to one form before comparing, and picks up timezone and date-format errors on the way. The gap it opens is silent: a field with no normaliser is compared raw, and nothing separates a field somebody decided needed no rule from a field somebody forgot.

The gap that decides the recommendation is the one between normalisation and constraint predicates, and it decides it because the two catch disjoint things. Normalisation asks whether the predicted argument equals the labelled one once both are canonical, so it needs a single right answer and rejects anything differing from it. A constraint predicate asks whether the argument has a property — start inside the requested window, attendees ⊇ required — so it accepts a whole set of values and needs no right answer. Neither contains the other: an argument can be acceptable without equalling the label, which normalisation rejects and a predicate passes, and an argument can satisfy every predicate while creeping toward one edge of the allowed window, which the predicate passes by construction. Pairing them is two instruments aimed at two failure modes rather than belt and braces.

The pairing that holds up is normalisation plus constraint predicates for structured fields, with a judge reserved for the one or two free-text fields per tool where a human would have to read it. On the scheduling set that produced a second number alongside the first: tool name correct on 88.8% of rows, tool name and arguments both acceptable on 71.3%. The seventeen-point gap is a real defect surface that the original metric reported as clean, and more than half of it was one thing — the agent emitting UTC timestamps derived from the requester's local time without converting. Neither level is interesting alone; the gap is the finding, since every row inside it is a call the name-only number counted as a success. That most of it is one timezone defect says where to look first: bucket the argument failures by field before arguing about whether the level is acceptable.

def score_action(predicted, row) -> dict:
    valid = row["valid_actions"]
    names = {a["tool"] for a in valid}
    if predicted.tool not in names:
        kind = "forbidden" if predicted.tool in row["forbidden_actions"] else "wrong_tool"
        if predicted.tool is not None and None in names:
            kind = "tool_when_none_needed"
        if predicted.tool is None and names != {None}:
            kind = "none_when_tool_needed"
        return {"tool_ok": False, "args_ok": False, "kind": kind}

    if predicted.tool is None:
        return {"tool_ok": True, "args_ok": True, "kind": "ok_no_tool"}

    spec = next(a for a in valid if a["tool"] == predicted.tool)
    args_ok = all(pred(predicted.args, row["state"]) for pred in CONSTRAINTS[predicted.tool]) \
              and normalise(predicted.args) == normalise(spec.get("args", predicted.args))
    return {"tool_ok": True, "args_ok": args_ok,
            "kind": "ok" if args_ok else "right_tool_wrong_args"}

The kind field is what makes the report actionable, because the five categories have different owners. wrong_tool and tool_when_none_needed usually move with the tool descriptions in the prompt; right_tool_wrong_args moves with the schema and the examples; forbidden is a safety review; none_when_tool_needed is often the model being over-corrected by a prompt that told it to stop calling things.

A scored decision is not an executed step#

The metric scores a decision, and a decision is not an execution. A perfectly chosen list_availability call against an endpoint that returns 503 is still a failed step, which is the territory of the last chapter — chapter 3 of this topic, on injecting faults so that recovery becomes measurable — and of the observability work that captured the error in the first place, in topic 10.8 on spans and traces. A high tool-selection score does not establish that any run completed.

Parallel calls need a decision too, since a model that can emit several tool calls in one turn makes the unit of measurement ambiguous. Scoring the emitted set against the valid-action set, with a rule that every emitted call must be individually acceptable and the required ones must all be present, keeps the metric interpretable; scoring each call as its own row inflates the denominator with the easy members of a batch and makes the rate drift whenever the model changes how much it batches. Either convention works as long as it is fixed, and the one that breaks comparisons is switching between them, which happens quietly when a model upgrade starts batching calls that the previous one issued sequentially.

There is also a subtler question about what the agent was conditioned on when it made the decision. Scoring single steps from labelled states means replaying each state and asking for one action, and the states in the set came from traces where the prior actions were whatever the agent actually did. Feeding the agent a curated prefix and scoring the next action measures the policy under a distribution it will not see in production, in the same way that teacher forcing flatters a sequence model — errors do not accumulate, because you keep resetting the context to something clean. That is an analogy, and teacher forcing is the training technique of showing a model the true previous token at each position rather than the one it just predicted. The mechanism carries across exactly; the purpose does not, and that is where it breaks. Teacher forcing is a deliberate training-time choice, and the model is afterwards run on its own outputs; a curated prefix in an eval is an artefact of how the set was built, and no later stage corrects for it. The number is still worth having, since it localises a defect to a single decision in a way that whole-run scoring cannot, and it should be read as an upper bound on how the same policy behaves twelve steps into a run it has been steering itself.

Finally, a high tool-selection score is compatible with an agent that is ruinous to operate. The eleven redundant reads in the no-tool class cost around 1,400 tokens each; at 40,000 runs a day with a similar rate, that is a line on an invoice, and the widening availability loop from chapter 2 of this topic — where the agent kept re-searching a broader date range, one pass at a time, until a mutually free slot appeared — does not show up in this metric at all, because every individual call in it is a defensible choice.

What the rebuilt set bought was not the 88.8%, a worse-looking figure than the one it replaced, describing a harder population. It bought the 41.9%: a rate on a class of decision the agent was taking thousands of times a day and that no earlier report could contain, let alone score. The rule underneath holds wherever decisions are scored against labels: a metric is only as complete as the action space its labels admit, and the action a schema built from observed tool calls will never contain is the decision to call nothing. Write labels as sets of acceptable actions, put no-tool in the set, sample the states so the class exists in the denominator, and publish the classes beside the aggregate. What those runs cost, and how to report it so the expensive ones do not vanish into an average, is the next thing to measure.