Tool-block placement in the prefix

The blocks that make up a request — a preamble, the tools array, a system prompt, the messages — go out in some order, and that order is what decides whether two requests share anything at all. Chapter 1 established the mechanism: a provider matches forward from token zero and halts at the first difference, so whatever varies earliest ends reuse for everything behind it. The operational question that follows is one nobody can answer from reading their own assembler, which is where in the serialised byte sequence each piece of content actually sits, and how much of the prefix a given piece has standing behind it.

Two instruments answer different halves of that. A pinned fingerprint — the hash of the rendered prefix, compared between builds, which topic 2.15 covers under what silently invalidates a cache — tells you whether the bytes moved when you did not mean them to. An offset map tells you where in the sequence a given segment lives and therefore how much content sits behind it. The hash catches unintended drift and the map prices intended change, and a green hash sitting on top of a badly ordered map is the expensive combination, because everything looks stable and nothing is being reused.

Read the prefix as an ordered map, not as a config object#

The unit of reasoning that makes tool versioning tractable is a list of segments with cumulative offsets, produced from the same serialisation path that talks to the provider. Assembling it takes a small amount of code and it answers questions that no amount of reading the assembler will.

import json

def prefix_map(payload) -> list[tuple[str, int, int]]:
    """(segment label, start offset, token count) over the serialised request."""
    segments, offset = [], 0
    for label, blob in serialize_in_wire_order(payload):
        n = count_tokens(blob)
        segments.append((label, offset, n))
        offset += n
    return segments

def tool_offsets(payload) -> dict[str, int]:
    """Where each tool's serialised definition begins, in tokens."""
    offsets, cursor = {}, 0
    for tool in payload["tools"]:
        offsets[tool["name"]] = cursor
        cursor += count_tokens(json.dumps(tool, separators=(",", ":")))
    return offsets

Running the first function over a real payload prints the layout the provider sees. Running the second prints the number that prices any future edit to a given tool, because the tokens behind a tool's start offset are the tokens that edit will invalidate. On a thirty-tool onboarding-support catalogue whose serialised prefix runs to 65,840 tokens, escalate_to_human — the tool most likely to have its description rewritten, since it encodes a policy that product keeps revising — sits at offset 3,940, which means rewording it costs 61,900 tokens of invalidation. Nobody on that team could have guessed that number, and once it is printed in a build artefact it is the kind of thing that gets quoted in review.

That figure is one subtraction and it is worth doing rather than accepting, because doing it is what tells you which of the two inputs to argue about. The whole prefix runs to sixty-five thousand eight hundred and forty tokens. The tool starts three thousand nine hundred and forty tokens into it, and everything in front of that offset survives the edit because the forward match has not reached the change yet. Subtract the survivor from the whole and the sixty-one thousand nine hundred is what is left behind the tool. Notice what does not appear anywhere in that calculation: the size of the edit. Rewording one sentence of escalate_to_human's description and rewriting the entire tool cost the same, because the term that would carry the difference has cancelled — both edits move the same start offset, and the loss is measured from the offset rather than from the edit.

Both inputs are properties of the layout, which is why an offset map is worth building at all. The total is decided by how much you cache; the start offset is decided by where the tool sits. Move a tool later and the offset rises and the loss falls, one for one, with nothing else in the request changing. Placement is the only lever there is, and it is a lever only because the subtraction has exactly two terms.

The map answers a second question at the same time, and it decides whether any of the tool offsets matter at all: the offset of the first segment that varies between two requests is the ceiling on everything, because no tool sitting behind it is cached, so its own offset prices nothing. Render the map for two different users and diff the two rather than rendering it once, since a single render cannot tell a segment that is stable from a segment that happened to be the same on the render you looked at.

The one function worth writing carefully is serialize_in_wire_order, because guessing at wire order is how prefixes end up in the wrong sequence in the first place. The reliable way to establish it is to capture the actual HTTP body once — most SDKs expose a request hook, a raw-response mode or a proxy setting for this, and a single captured payload written to disk settles the question permanently for that client version. Read it top to bottom and label the segments in the order they appear, rather than in the order your assembler builds them, since the two are related only by convention. That capture is also the thing to re-run after an SDK upgrade, because the order a client emits fields in is not usually covered by its compatibility promises.

What you can reorder, and what the API decides for you#

Ordering advice only helps to the extent that the position is yours to choose, and that varies by stack in ways that decide how much of this chapter you can act on.

The first three rows collapse to the same operational fact: on a hosted API you do not get to hide a churny tools array behind a stable 60,000-token document, so the tools block is structurally the most expensive place in the request to make a change. That is the reason this topic exists as something separate from general cache hygiene. Every other block in your prompt can be pushed backward to limit its blast radius; the tools block is pinned at the front and the blast radius is everything.

The last row is the exception that clarifies the rule. When you own the chat template — a self-hosted model behind vLLM with a radix prefix cache, which indexes stored prefixes by the leading tokens they share, for example — you can render tool definitions after the static policy text, and a description edit then invalidates only the few thousand tokens of tool block that follow it. Teams running their own inference sometimes do this deliberately, and it turns a 61,900-token event into a 1,900-token one. Reaching for it on a hosted API by shoving tool documentation into the system prompt as prose is a bad trade, because you lose structured tool calling to save a cache fill.

The gap between those two figures is worth reading as a quantity rather than as a contrast, because it names exactly what was bought. Nothing about the edit changed and nothing about the tool changed. What changed is which blocks sit behind the start offset: with the policy text rendered ahead of the tools, the only content still standing behind escalate_to_human is the remainder of the tool array, and the difference between the large number and the small one is the static document that swapped sides. Every token of stable content you can get in front of a volatile segment comes out of the loss, and every token you leave behind it stays in.

Sorting the array is a change at position zero#

The tidiness suggestion — alphabetise the tools array while you are in there — is worth taking seriously as a design question rather than dismissing, because it comes from a real instinct. A deterministic order is genuinely valuable; an array whose order depends on dictionary iteration or plugin registration is exactly the serialisation drift that takes hit rates down overnight after a dependency bump. The instinct to pin the order is right. The mistake is which order and when.

Alphabetising a thirty-tool array typically moves twenty-nine of the thirty definitions, and the first one it moves is at or near offset zero, so the change invalidates the entire prefix for every request in the fleet. Bundled into a pull request labelled "fix typo in search_shipments description", it converts a tail invalidation into a full one, and it does so invisibly, since a reviewer reading the diff sees the same set of tools before and after. A reviewer reading the offset map sees thirty entries whose offsets all changed.

The order that serves you is by expected edit frequency, most stable first, pinned in an explicit list rather than inferred from any registry's iteration order.

# Order is part of the cache key. Append new tools at the end; do not sort.
TOOL_ORDER = [
    "get_order",           # stable since launch
    "get_customer",
    "search_shipments",
    # ... 26 more, stable ...
    "escalate_to_human",   # description revised roughly monthly
    "draft_refund_note",   # newest, still settling
]

def serialize_tools(registry) -> list[dict]:
    return [registry[name].to_schema() for name in TOOL_ORDER]

Putting the volatile tools last means a monthly rewrite of escalate_to_human invalidates only the one or two tool definitions that still follow it, plus the system prompt and policy document, instead of the 61,900 tokens it costs from its current offset — of which 1,900 is tool schemas that had no reason to be behind it. The saving is real and it is also modest, since the policy document dominates the total either way. The larger reason to pin the list is that it makes order a reviewable decision. A new tool appended at the end costs the tail; a new tool inserted in the middle because it reads better next to its sibling costs everything from there down, and with an explicit list that insertion shows up as a moved line rather than as nothing at all.

Per-request tool sets split the cache#

No ordering makes a per-request tool set cacheable. If half your traffic gets an extra tool because of a feature flag, or a tenant's plan grants three tools that others do not have, the tools block differs at the front for those populations and you have split your cache rather than lost it — each population caches fine against its own variant, and the entries divide accordingly. The remedy is to shrink the number of variants, by sending the union of tools with authorisation enforced in the handler rather than in the schema, and that is a security and prompt-clarity trade-off rather than a caching one.

A catalogue assembled at runtime from MCP servers is the version of that problem where the order is not even yours. Both the set and the sequence can change without a deploy on your side — a server that adds a tool, or returns its list in a different order after a restart, has moved your offsets while your own repository shows no diff at all. The pinned list from the section above applies one level out: hold an explicit order over the names you expect, sort unknown arrivals to the end rather than wherever the protocol put them, and treat a change in the set as a schema change with a cache cost attached.

Paying the write premium on an entry nobody reads is worse than not caching at all, which is the part that makes a badly ordered prefix expensive rather than merely wasteful. Cache writes are billed above the ordinary input rate and cache reads well below it, and the arrangement only comes out ahead across the reads. An endpoint whose divergence point sits below the minimum cacheable length never reaches the reads, so it collects the surcharge on every request and the discount on none, and the bill is worse than the same traffic would have produced with caching switched off. A hit rate that never rose above a couple of percent is therefore not a missed opportunity. It is a charge being paid for nothing.

Moving a preamble backward does not make it free, either. Forty tokens of identity still get prefilled at the full input rate on every request, wherever they sit; what moved was the 65,800 tokens behind them. Nor does the move come without a quality question, since instructions placed after a 60,000-token document are read at a different position in context than instructions placed before it, and the placement guidance from topic 4.14, on resolving the conflict between position bias and prefix stability, still applies. A forty-token preamble is identity the model needs available when it reasons about the user's request, so the last message is the right home for it; a 400-token behavioural instruction that materially changes tool selection may deserve to stay high in the prompt and be paid for.

The offset map now tells you what any edit costs before you make it. What it does not tell you is whether the edit needs to happen at all — and the question of whether a proposed change is even visible to the model turns out to have a different answer, surprisingly often, from whether it is visible to the cache.