Tool-call assertions

An agent's final message is written after the call it describes, by the same component that made it, from memory. That makes it the worst available evidence about what the agent did and the most convenient to assert on, because it is a string and every test framework knows what to do with one. A tool call here means the model's request that the harness execute one named function with named arguments, recorded as the provider emitted it — the arguments_raw field of the run record from the previous chapter, kept before any handler saw it.

This chapter writes the assertion that reads that field, and the data format that keeps two hundred such cases honest while fourteen tool signatures move underneath them. The assertion is four claims long and each fails with a different diagnosis, which is the reason to write them separately rather than as one composite condition.

The assertion, precisely#

These five checks were live on a refund agent, all reasonable, all passing, while nine days of $0.42 refunds went out:

Read down the third column and the pattern is that four of the five graded the agent's description of what it did, and the fifth graded that it did something. The number that reached the payments API appears in none of them.

Each of those checks is evidence about something, and what it is evidence about is decided by where in the run the artefact it reads was produced. The final message is produced last, by the model, after the call has already been made, and from the model's own account of making it — so a check on the message is evidence about that account and nothing else. The handler's clean execution is produced after the dispatcher has coerced the arguments into whatever the signature wanted, so it is evidence about the dispatcher's willingness to accept what it was handed. An assertion can only speak about the stretch of the run between the moment its artefact came into existence and the moment that artefact was captured, and none of those five stretches contains the instant at which the model chose a number. The recorded call is the one artefact that does: it is created by the model at the moment of choosing and captured before anything could rewrite it.

def assert_refund(record: RunRecord, case) -> None:
    calls = [c for c in record.tool_calls if c.name in SIDE_EFFECTING]

    # 1. exactly one side-effecting call, and it is the right tool
    assert len(calls) == 1, f"expected 1 side-effecting call, got {[c.name for c in calls]}"
    call = calls[0]
    assert call.name == "issue_refund", f"called {call.name}"

    # 2. arguments, read as the model emitted them
    args = json.loads(call.arguments_raw)
    assert args["order_id"] == case.order_id
    assert args["amount_cents"] == case.expected_amount_cents

    # 3. the type, not only the value
    amount = args["amount_cents"]
    assert isinstance(amount, int) and not isinstance(amount, bool), \
        f"amount_cents was {type(amount).__name__}: {amount!r}"

    # 4. nothing else with a side effect ran
    assert set(args) == {"order_id", "amount_cents"}, f"unexpected args: {sorted(args)}"

Claim three is the one that looks like paranoia and is not. In Python 4200 == 4200.0 is true, and True == 1 is true, so an equality assertion alone accepts a float where the API expects minor units as an integer, and accepts a boolean anywhere the expected value is 1. A model that emits {"amount_cents": 4200.0} will pass the value check, pass the handler's int() coercion, and send 4200 — which is fine today and stops being fine the moment a different field is a quantity where 0.5 rounds to 0. Asserting the type at the point where the model chose it costs one line and pins the thing the schema on the wire could not.

That last phrase is worth unpacking, because the belief it contradicts is the usual reason a team concludes that tool-call assertions are redundant. A tool's JSON schema is a predicate over the shape of an argument object: which keys must be present, what type each value has, sometimes a range or a list of permitted strings. It is written once, for the tool, and it is evaluated with no knowledge of the case in front of it. The object {"order_id": "ORD-448192", "amount_cents": 42} satisfies a schema requiring an integer amount_cents, because 42 is an integer, and it would satisfy any tighter schema anybody could reasonably write, because 42 is also positive, also below any sane upper bound, and also a plausible number of cents. Constrained decoding — the technique of restricting generation so that only schema-valid tokens can be emitted, which the second chapter of this topic covers — makes that guarantee certain rather than likely, and leaves its subject exactly where it was.

The reason no schema catches this is structural rather than a failure of care. The correct value of amount_cents is a relation between the call and one particular case: forty-two dollars for this mug, on this order, under this refund policy. A schema attached to the tool is the same for every call the tool will ever receive, so it has no term in it that could refer to the case. A constraint that cannot mention the case cannot decide a property that depends on it. The check must therefore live where the case's expected value lives, which is the whole reason the expectations in the next section sit in data beside each case rather than in the tool definition.

The same argument applies one level up, to which tool was called at all. A schema governs the arguments of a tool once that tool has been selected; nothing in it speaks about whether selecting it was right. An agent that calls cancel_order with a well-formed order id has emitted a perfectly schema-valid call, and the customer wanted a partial refund.

Reading the arguments from arguments_raw rather than from the parsed dictionary the handler received is what makes claim three meaningful at all. Parse the raw text inside the assertion, with no model and no coercion between the provider's bytes and the comparison. If you instead read call.arguments_received, you are asserting about the output of your own coercion layer, which will have already turned "4200" into 4200 and told you nothing about what the model chose. Both are recorded, and the difference between them is itself worth a check on any tool where types have consequences — a mismatch means your dispatcher is rewriting the model's intent, which is useful to know before it rewrites it in the other direction.

Claim four is the guard against a class the other three miss entirely. An agent that calls issue_refund correctly and also calls cancel_order has done the right thing and one wrong thing, and a check written as "the refund call is correct" is silent about the second. Partitioning tools into read-only and side-effecting is what makes that assertable in one line, and the partition has to be a declared property of each tool rather than a judgement made per case:

SIDE_EFFECTING = {"issue_refund", "cancel_order", "update_address",
                  "send_email", "create_ticket"}
READ_ONLY = {"lookup_order", "search_policy", "get_customer", "list_shipments"}

Keeping the two sets exhaustive of the registered tools — and asserting that at suite startup — means a newly added tool cannot silently join the read-only side by omission. That single line of housekeeping is what stops the check degrading as the agent grows.

The four claims are not four degrees of the same check. Two of them ask which action the agent took — how many side-effecting calls there were, and which tool they named. One asks what the chosen action was applied to. One asks how the model expressed the value it chose. A claim about which action was taken implies nothing about the operands it was applied to, and every combination of the two occurs. The right tool with the right arguments is the pass. The right tool with a wrong argument is nine days of $0.42 refunds. The wrong tool with arguments that are individually impeccable is the cancel_order case above, where every value is correct for the call that was made and the call was not the one anybody wanted. The wrong tool with wrong arguments is the combination even a loosely written check tends to catch, and it is not the one that reaches production.

Grade the action and the operands as separate claims and each failure arrives already diagnosed, which is the practical return on writing four asserts instead of one condition. A wrong call count usually means the loop retried, or that the prompt invited a belt-and-braces second action. A wrong tool name is a selection failure, and selection is driven by the tool descriptions far more than by the system prompt, so that is where the fix goes. A wrong argument value means the model read the case wrongly — the order, the amount, the policy — which is a context or prompt problem. A wrong type means the value was chosen correctly and rendered badly, which is a schema and description problem again. Fused into one composite condition, all four arrive as the same red line, and whoever is reading it at the time has to work out which of four unrelated bugs they are looking at before they can begin.

Two hundred cases and a signature that keeps moving#

Written inline, that assertion is fine once and unmaintainable at scale. A suite with 200 tool-bearing cases and fourteen tools accumulates 200 hand-written expectation blocks, and a tool signature changes more often than anyone expects — renaming amount_cents to amount_minor_units, adding an optional reason_code, splitting update_address into two calls. Nine such changes in six months is an unremarkable rate for an agent under active development, and each one is a chance for the suite to rot in the worst possible direction.

The structure that holds up has three parts. Expectations live in data beside the case rather than in code:

- case_id: refund_partial_damaged
  input: "the mug arrived cracked, I want money back for it"
  expect_tool_calls:
    - name: issue_refund
      args:
        order_id: ORD-448192
        amount_cents: 4200
      arg_types:
        amount_cents: int
  forbid_other_side_effects: true

One generic checker reads that data and does the work, so there is exactly one place where "how a tool call is graded" is defined. And a contract check runs once per suite, before any case, validating every expectation against the live tool definitions:

def validate_expectations(cases, tool_schemas) -> None:
    for case in cases:
        for exp in case.expect_tool_calls:
            schema = tool_schemas.get(exp.name)
            assert schema, f"{case.case_id}: unknown tool {exp.name!r}"
            params = set(schema["input_schema"]["properties"])
            unknown = set(exp.args) - params
            assert not unknown, (
                f"{case.case_id}: expects {sorted(unknown)} on {exp.name}, "
                f"which now takes {sorted(params)}")
            assert exp.args, f"{case.case_id}: empty expectation is vacuous"

The rename that would quietly break 200 cases now fails once, at startup, naming the tool, the old parameter and the new parameter list. A morning's work becomes a five-minute substitution, and the failure is loud rather than absent. The last line of that function guards the specific rot this structure invites: a checker written as all(args.get(k) == v for k, v in expected.items()) returns True for an empty expectation, so a case whose expectations were deleted during a refactor passes forever while asserting nothing.

There is a tension in how strict to be about the argument set. set(args) == {"order_id", "amount_cents"} fails the day someone adds an optional idempotency_key, which is a legitimate improvement and will annoy 200 cases at once. Comparing only the keys the case names — all(args[k] == v for k, v in expected.items()) plus the startup contract check — tolerates additions and stops noticing extras. The workable middle is to assert equality on the named arguments, assert that no required parameter of the current schema is missing, and let unnamed optional parameters through. That way the suite fails when the agent omits something the tool needs and does not fail when the tool grows a field nobody in the eval cares about.

Order is the other property worth expressing in that data format, and it needs a deliberate choice rather than a default. Some cases care that the agent looked the order up before refunding it, because refunding an order it never read is a correct-looking call made on an assumption; other cases care only that both happened. Encoding this as a list with an ordered: true flag keeps the strict version available without imposing it everywhere, and it keeps the checker honest about what it is asserting — a sequence comparison and a multiset comparison, the second of which counts how many times each call appears and ignores the order entirely, are different assertions and should not share a name. The transcript supports both, because the run record of the previous chapter preserved emission order and the call_id that pairs each result back to the call that asked for it.

Grading a tool call with a judge#

The tempting alternative, and the one that comes up in review almost every time, is to show a model the request and the response and ask whether the refund amount in the answer matches the amount requested. It is a smaller diff, it needs no transcript plumbing, and it sounds more general.

It fails at three separate points, and the three are worth separating because only one of them is about cost. The first is that it grades the wrong artefact. The judge reads "I've refunded $42.00", the request says $42, and the two agree; the answer is correct narration of an incorrect action, which is precisely the failure. To grade the call itself the judge would need the transcript — at which point it is being asked to compare two numbers that it has been handed, and a comparison operator does that with no prompt.

The second is that it introduces a stochastic verdict on a deterministic question. A judge that agrees with ground truth 97% of the time disagrees with it on three cases in every hundred, so over 200 cases it will flip roughly six on any given run, and it flips them in both directions — some passing runs marked failed, some failing runs marked passed. The six are not the same six twice. The judge's verdict is itself sampled from a model, so the cases that flip are the ones sitting near its own decision boundary, and each of those is decided afresh every run with no memory of the last one. A suite whose failures are irreproducible stops being used as a gate, because the first thing anyone does with a red build they cannot reproduce is re-run it. The rate also cuts the wrong way exactly where the gate is supposed to earn its keep: in a suite that is usually green, genuine regressions are rarer than three cases in a hundred, so most of the red a judge produces is its own noise rather than a finding, and a team works that out within a fortnight and starts reading red as noise.

The third is cost and latency, which are the least important of the three and the easiest to quantify. Take a judge call on a 1,800-token transcript with a 200-token verdict, priced at $3 per million input and $15 per million output tokens. Prices quoted per million tokens become prices per token by dividing by a million, so the input side is 1,800 millionths of $3, a little over half a cent, and the output side is 200 millionths of $15, about three tenths of a cent. Together that is roughly $0.0084 for one case. Multiplying by the suite's 200 cases is allowed because each case pays independently and nothing is shared between them, and it gives $1.68 per suite run. A team merging thirty pull requests a day runs the suite thirty times, so the daily figure is roughly $50, and each case now waits on a model call it did not previously make, adding about 1.4 seconds of latency, for arithmetic that == performs in nanoseconds.

There is real work for a judge, and none of it is here. It belongs where the property being graded has no closed form — whether an explanation is faithful to a retrieved document, whether a tone is appropriate, whether a refusal was warranted. 4200 == 4200 has a closed form, and paying a model to evaluate it buys a slower, more expensive, occasionally wrong version of an operator you already have.

Stubbed side effects and unwarranted calls#

A correct tool-call assertion does not tell you the side effect happened. The transcript records that the agent asked for issue_refund(order_id="ORD-448192", amount_cents=4200), and in an eval run the handler is usually a stub that returns a canned success. Whether the real payments API would have accepted that call, whether it was idempotent — safe to submit twice without paying the customer twice — whether it was applied twice on retry: those are integration concerns, and an eval suite that stubs the world cannot speak to them. Nothing here replaces a contract test against the payments service.

That limit follows from where the artefact was captured, which is the same principle that made the assertion trustworthy in the first place. The recorded call is created at the moment the model chooses and captured before the dispatcher touches it, so it carries evidence about everything upstream of that point and none about anything downstream of it. A green suite over stubbed handlers is therefore asserting a conditional: if the payments service behaves as the stub does, this run would have refunded $42.00. The suite never evaluates the antecedent, and the antecedent is where contract tests live. Reading the conditional as its consequent is how a team ends up with two hundred green cases and a refund endpoint that has quietly started requiring an idempotency key.

A correct tool-call assertion also does not tell you the tool was the right thing to do. Asserting that issue_refund was called with the case's expected arguments presumes the case is right that a refund was warranted. A model that refunds enthusiastically will pass every refund case in the suite and fail the business; catching that needs cases where the correct behaviour is no side-effecting call at all, which the same machinery expresses as an empty expectation with forbid_other_side_effects: true — and which is why the vacuity guard above has to distinguish "expects nothing" from "expects that nothing happens".

Grading what the agent did rather than what it said about it is a change of artefact, not a change of strictness, and it is available only because the run record kept the provider's bytes. The four claims are cheap enough to write for every tool-bearing case and specific enough that each failure names its own cause: the wrong tool, the wrong order, the wrong amount, the wrong type, or a second call nobody asked for. Keeping them in data with a startup contract check is what makes two hundred of them survive a year of signature changes.

None of it tells you the run was affordable. The suite can be green on every argument of every call while a prompt change has tripled what each case costs to produce, because nothing written so far reads the usage block the record went to the trouble of keeping.