Reading the serialised tool payload

Between the tool definition in your source tree and the tool definition on the wire sit an SDK, usually a framework, sometimes a decorator that reflects over type hints, and occasionally a piece of middleware that appends its own instructions. Each of those is a transformation, and none of them promises to be faithful. The result is a request body that may not contain what you believe you wrote, and since that body is the entire input to the model's choice, a discrepancy there makes every downstream theory about model behaviour unfalsifiable.

Reading the serialised payload is the habit that closes this off. It costs about ninety seconds, it requires no instrumentation of the model, and it converts the open-ended complaint that a tool is being ignored into a narrow question with a small set of answers: given that the model saw exactly this, why did it choose otherwise. Everything in the next chapter assumes you have done it first.

Print the array before you theorise#

Most SDKs will hand you the exact bytes if you ask. With the Anthropic Python client the underlying HTTP client is exposed, so an event hook logs the request body as it is sent, after every layer of serialisation has run:

import json, httpx, anthropic

def dump_request(request: httpx.Request) -> None:
    if not request.url.path.endswith("/messages"):
        return
    body = json.loads(request.content)
    print(json.dumps(body.get("tools", []), indent=2))
    print("system:", json.dumps(body.get("system"))[:400])
    print("tool_choice:", body.get("tool_choice"))
    print("messages:", len(body.get("messages", [])))

client = anthropic.Anthropic(
    http_client=httpx.Client(event_hooks={"request": [dump_request]})
)

An equivalent exists everywhere. OpenAI's clients let you pass a custom http_client in the same way; LangChain and LlamaIndex both expose the rendered tool payload through their callback handlers; if the framework is opaque enough that neither works, pointing base_url at a local reverse proxy for one run always works, because the request is an HTTP request no matter how many abstractions produced it. Whichever route you take, capture the payload from a real production code path rather than from a test that builds the toolset by hand — the discrepancy you are hunting is often in the assembly code that only production runs.

One caution about where the payload is captured, since it is the step people skip. Test fixtures and notebooks tend to construct the toolset directly, which bypasses exactly the layers under suspicion: the feature-flag filter, the router that trims the array, the decorator that reads docstrings off your functions. A payload dumped from that path is a dump of your intentions rather than of the request, and it will look correct in precisely the cases where production is wrong.

Once you have it, four things in the dump are worth reading closely, and the first is simply whether the tool is there at all. Toolsets are frequently filtered by feature flag, by tenant plan, or by a router that trims the array to the six tools it thinks are relevant, and a tool absent from the array cannot be selected by any amount of prompt engineering. After that: the description, in full, exactly as serialised. Then the schema, field by field, including which names ended up in required. Then everything around the array — the system prompt, the tool choice setting, and the message history.

The projection from your repository to the wire#

The useful mental model is a compiler with a very narrow output format. Your function has a name, a signature, type annotations, a docstring, validation logic, comments explaining why the second parameter exists, an owner in CODEOWNERS, and tests. Serialisation projects a small subset of that onto the wire, and everything outside the subset is invisible to the model in the strict sense that it was never transmitted.

The comparison to a compiler is a picture, and it is worth saying where the picture misleads. A compiler that cannot represent something in its output format stops and tells you: the type does not fit, the name is not in scope, here is the line. This projection has no such step. A docstring longer than the serialiser reads, an enum the derivation does not recognise, a default it cannot express — each of those produces a request body that is entirely valid, that the provider accepts without complaint, and that omits the thing you cared about. Silence is the normal outcome of a lossy projection here, which is why the loss has to be found by looking at the output rather than by waiting to be told.

Derived schemas are where the projection loses most of what people assume survives. A Section enum in Python becomes {"type": "string"} under some serialisers and {"type": "string", "enum": [...]} under others, depending on the library version. A default value in the signature may become a JSON Schema default keyword, which providers do not enforce and models treat as a hint at best, or it may vanish, or — the case that causes real damage — the field may still appear in required because the derivation used the annotation rather than the default to decide. A Union[str, None] becomes anyOf with a null branch, which reads to a model as permission to send null rather than as permission to omit. None of these are bugs in your function; they are what happened at the boundary.

The concrete version of that last case is worth walking through, because it produces a symptom nobody attributes to serialisation. A function declared as def search_docs(query: str, section: str | None = None) reads unambiguously in Python: section has a default, so callers may leave it out. One widely used derivation path builds the properties from the annotations and the required list from the fields without a NotRequired marker, and str | None is not that marker, so section lands in required with a nullable type. The model, obeying the schema it was given, sends "section": null on every call. Your function then receives None, which is exactly the default, and everything appears to work — until someone adds a validator that rejects nulls, or until you notice that the model has started sending "section": "billing" on questions that are not about billing, because a required field invites a value. The Python signature and the wire schema disagreed the whole time, and only one of them was ever in the request.

"A required field invites a value" is doing real work in that sentence, and it is not a figure of speech. A field in required cannot be omitted: the decoder will not allow the object to close until the key has appeared, so on every call the model reaches a position where it must write something for section. Given a nullable type it may write null, and given a conversation that mentions billing at all it may write "billing", because both satisfy the schema and one of them looks more useful. Two symptoms with one cause: the nulls and the spurious values are the same constraint resolved two different ways on two different turns. Neither would occur if the field had simply been absent from required, which is what the Python signature said and what nothing on the wire repeated.

The belief this defeats is that the type annotation is the contract. In your codebase it is — mypy checks it, the tests exercise it, and a caller who leaves section out gets exactly the behaviour the signature promises. None of that machinery runs on the provider's side, and none of its conclusions travel. The only contract in force is the one serialised into input_schema, and the derivation that produced it is a third-party function whose behaviour across versions nobody on your team has agreed to. Checking the annotation tells you what you meant. Checking the payload tells you what was said.

Deriving schemas from code is still the right default, because a hand-maintained JSON blob and a function signature drift apart within a month and nothing tells you when they do. The discipline that makes derivation safe is asserting on the output: build the payload in a test, and compare the serialised tool array against a checked-in fixture, so that an SDK upgrade which quietly changes how enums or optionals are rendered fails in CI rather than in production selection rates. Teams that do this find the failure the day it lands, and teams that do not find it in a support metric three weeks later.

The array is not the whole of what the model reads#

Selection depends on the entire request, and the array is one part of it. A system prompt saying "answer the user's question directly and concisely, without unnecessary steps" is competing with every tool description you wrote, and it usually wins, because it is stated as a general instruction while the tool description is stated as a conditional one. Contradictions of that shape are common in codebases where the prompt was written before tools were added and never revisited.

The step swallowed in that "because" is worth unpacking, since it is the reason the contradiction resolves the way it does rather than the other way. Both sentences are in the same request and neither is marked as outranking the other; there is no precedence rule in the protocol, so what settles it is which instruction applies to the turn in front of the model. "Answer directly and concisely, without unnecessary steps" applies to every turn, with no test to evaluate — it is in force by the time the model has read the request. "Use this when the customer has a documentation question" applies only once the model has decided that this turn is such a case, which is a judgement that can go either way. An instruction that is unconditionally in force and one that must first be argued into force are not competing on equal terms. That is also why the repair in this chapter's scenario worked: replacing the general clause with a general clause pointing the other way changed the instruction that applies every time, rather than adding another conditional one.

Notice what that implies about where a fix belongs. If the general clause is wrong, no amount of strengthening a tool description will outweigh it, because strengthening a conditional instruction does not make it unconditional. Emphatic capitals in a description are an attempt to win an argument the description is not in.

Tool descriptions and the system prompt are therefore better treated as one document with two storage locations. Anything that applies to a single tool belongs in that tool's description, where it sits next to the schema the model is filling in; anything about how the assistant should weigh looking things up against answering belongs in the system prompt, stated once. Splitting a rule across both — "use tools when appropriate" in the prompt, "use this when appropriate" in each description — produces a request in which nothing says anything specific, which is the state most underperforming toolsets are actually in.

The message history counts as well, in a way that is easy to underestimate. Every earlier tool call in the conversation is a worked example of how these tools get used, and every earlier tool result is evidence about what they return. Three consecutive search_docs results that came back empty are three demonstrations that the tool does not help, and the model's behaviour after them reflects that. If your dump shows a history full of {"results": []}, the selection problem may be a retrieval problem wearing a costume.

The step under that is the same one that makes tool renaming unsafe. A request is a single sequence, and the model has no partition of it into declarations it must obey and transcript it may learn from. A description asserts that search_docs finds documentation; three results in the same sequence show it returning nothing. Both are text in front of the model, and demonstrated outcomes are a stronger signal than stated intentions for a system trained on sequences. So the behaviour degrades gradually within a session rather than switching off, and it degrades in a way no change to the description will reverse, because the description is not the part that lost the argument. The diagnostic follows directly: when selection is fine early in a conversation and poor later, look at the results in the history rather than at the array.

The tool choice setting, which chapter 6 covers in earnest, sits in the same payload and is worth glancing at during any selection investigation for a plain reason: a value of none disables calling entirely, and a value forcing a different tool makes the tool you are worried about unreachable. Both are usually set by code far from where the tools are declared — a summarisation path, a moderation pre-pass, a retry wrapper — and both explain a selection rate of exactly zero.

Selection is a sample, not a lookup#

Reading the request tells you the input to the decision. It does not tell you the decision, and it never will, because selection is a sample from a distribution. The same payload sent twice at temperature 1.0, the sampling setting that draws from the model's own distribution without sharpening it, can produce a call one time and prose the next, so a single trace showing a reasonable payload and a missed call is not evidence that the payload is fine — you need the rate over a set of conversations before you can say anything, and twenty replayed conversations is usually enough to distinguish "never selects" from "selects two thirds of the time".

The reason twenty is enough is arithmetic rather than convention, and it is worth doing once so that you can pick your own number next time. Treat each replayed conversation as an independent draw. If the true selection rate were two thirds, the chance of a single conversation not producing a call is one third, and the chance of twenty in a row not producing one is one third multiplied by itself twenty times — something like one chance in three and a half billion. So twenty silent replays are not weak evidence against a two-thirds rate; they are conclusive. The same sum run the other way is why one trace proves nothing: a single miss against a two-thirds rate happens one time in three, which is to say constantly. What twenty replays cannot do is separate two thirds from three quarters, and if that is the comparison you are making you need a great many more conversations than twenty.

The selection figures in this chapter's scenario should be read as rates in that sense and not as a score. Documentation-question selection moved from 6% to 71%, which is close to twelve times as many questions being looked up, on the same replay set with the same four-word description in place. Holding the description fixed is what makes the comparison mean anything: one thing changed, so the difference is attributable to it. Had both the description and the system prompt been edited in the same deployment, the figures would have measured a bundle, and the team would still not know which of the two had been worth their week.

There is also a layer below the payload that you cannot see. The provider transforms your tools array into whatever internal representation the model was trained on, and that transformation may include boilerplate instructions about how to use tools that you never wrote and cannot inspect. This is why the same tool array can behave differently across model families and across versions of the same family: the definition is identical and the rendering of it is not. It also means the token count you compute locally for a tool definition is an estimate rather than the billed figure, though in practice it lands within a few percent.

Finally, the payload says nothing about whether the tool works. A definition can be perfect, the model can select it correctly on every appropriate turn, and the feature can still be broken because the function returns an empty list, or times out, or returns 700 tokens of internal warehouse codes the model cannot interpret. Payload inspection answers what the model was working from, which is the question that stops the guessing.

The habit is worth stating as a rule, because its value is almost entirely in being applied before rather than after. Any claim about why a model chose or did not choose a tool is a claim about the contents of a request body, and that body is available to you, in full, for the cost of one event hook. A team that reads it first spends its time on the description, the schema and the prompt that are actually in flight; a team that reads it fifth spends three days rewriting a docstring that nothing transmits.

What the payload hands you once it is confirmed correct is a much narrower question, and it has a small set of answers: given that the model saw exactly this, why did it choose otherwise. The next chapter works through them.