Examples in a tool description

The search_issues tool in an internal engineering assistant had a description that ended with a worked example, because the query language it wrapped was fiddly and the example made calls come out right:

{
  "name": "search_issues",
  "description": "Search the issue tracker. Accepts a query in the tracker's filter syntax and returns matching issues with id, title, state, assignee and updated_at. Example: query=\"status:open AND assignee:me\"",
  "input_schema": {
    "type": "object",
    "properties": {
      "query": { "type": "string", "description": "Filter expression." },
      "limit": { "type": "integer", "description": "Max results, 1-100. Defaults to 25." }
    },
    "required": ["query"]
  }
}

Then the tracker team shipped a new parser. Terms became conjunctive by default, AND stopped being a keyword, and an unrecognised bare word became a hard error rather than a free-text match. Two weeks later the assistant's error rate on issue search sat at 91%, and every failing call looked like this:

{"name": "search_issues", "input": {"query": "status:open AND assignee:alice@corp.dev", "limit": 25}}

and came back with:

{"type": "error", "error": {"code": "parse_error", "message": "unknown token 'AND' at position 12"}}

The routing was perfect. Out of 1,240 issue-search requests that fortnight, the model chose search_issues correctly on 1,238 of them, and then produced a query in a syntax that had not existed for a month. The tool was selected right and called wrong, every time.

Why the example outranks the type#

The schema said query is a string. That is true, it was still true after the migration, and it constrains nothing, because every possible query — old syntax, new syntax, nonsense — is a string. Against that void of guidance sits a concrete, well-formed instance of exactly the thing the model is about to produce, sitting a few hundred tokens up the context. The model does what models do with a demonstration: it completes the pattern.

Those two artefacts are not competing on equal terms, and the reason is that they act at different points in the process. A schema is a filter applied to the argument object after the model has produced it: your agent loop validates the object, and a failure goes back as an error. Nothing about a schema shapes the tokens while they are being generated, beyond whatever the model takes from reading it as text like any other. An example is the opposite kind of thing. It is text sitting in the context at generation time, and conditioning on the text in the context is the entire mechanism by which the next token gets chosen. So the schema can only reject what the example has already caused, and here it had nothing to reject, because a query in the retired dialect and a query in the current one are equally strings.

This is few-shot prompting — showing the model a handful of worked instances of the task rather than describing the task in the abstract, and letting it complete the pattern — arriving through a field that does not look like a prompt. A description that ends with Example: query="status:open AND assignee:me" is functionally the same object as a one-shot example in a system prompt, and it behaves the same way: it fixes the shape of the output far more tightly than any abstract description of that shape. That is precisely why the example was worth adding in the first place. Before it existed, the model invented plausible-looking filter dialects; after it existed, calls came out in the house syntax. Its power and its failure mode are the same property.

The strength of the effect is easy to underestimate, because it survives contradiction. The assistant's owner first tried to fix the problem by adding a sentence to the front of the description — The tracker no longer supports the AND keyword; terms are implicitly ANDed — and leaving the trailing example in place. Over the next 200 issue-search calls, 6 used the new syntax. The instruction was read, was unambiguous, and lost to the demonstration sitting after it. Deleting the example and replacing it with a current one moved the same 200-call replay to 189 correct. An abstract rule competes badly against a concrete instance of the thing being generated; the instance is what the next-token distribution is shaped around.

Position within the string moves the effect in the direction that observation predicts. The same corrective sentence placed after the example rather than before it did better and still lost, at 63 of 200 calls in the new syntax against 6 with it in front, and 189 once the example itself was replaced. Labelling the old example in place — Example (deprecated syntax): query="status:open AND assignee:me" — scored 41, which is the version most people try first because it preserves the audit trail. A demonstration is not neutralised by a caveat attached to it, since what conditions the output is the token sequence itself and the sequence still contains a well-formed call in the old dialect.

There is a pattern across those four replays, and naming it predicts what will happen the next time somebody attempts the careful fix. A corrective sentence does better the closer it sits to the point of generation, which is why placing it after the example beats placing it before. It never wins outright, because proximity is not the only variable: a demonstration is a well-formed instance of the exact object about to be produced, and a sentence about that object is not the same kind of thing at all. Labelling the example lands between the two, since the label is adjacent to the demonstration while the demonstration itself is still intact and still well-formed. Only replacement moves the number properly, because only replacement changes what the well-formed instance in the context is an instance of.

The limit row is the same mechanism read the other way. Nothing in the description discouraged the model from paging, and the schema documented the full range, but the example modelled a call with one argument, so the model produced calls with one argument or with limit pinned to the number the prose mentioned. Whatever your examples happen to show becomes the default behaviour of the tool, including the parts of them you did not think of as choices.

A current example can still be the wrong example#

The stale-syntax failure is loud, which is most of what made it survivable: 91% of calls erroring is a graph anyone can see. The same mechanism produces a quieter failure with an example that is completely correct. After the migration was cleaned up, search_issues carried a current demonstration, query="status:open assignee:me", and the tool worked. Over the following six weeks it took 1,190 calls, of which 11 contained a date term. A hand-labelled sample of 200 of the requests behind those calls found 44 that were explicitly time-bounded — what did we close last week, is anything still open from the Q1 push, did that regression get filed before or after the release — so roughly a fifth of the traffic wanted a filter that the model used on under 1% of calls.

Forty-four in two hundred is a little over a fifth; eleven in one thousand one hundred and ninety is under one in a hundred; so the gap between what the traffic wanted and what the tool was asked for is better than twenty to one. That ratio is the measurement, and it only exists because somebody labelled requests rather than calls. Calls tell you what the model did, and every one of these calls succeeded. Requests tell you what was asked. The difference between the two is exactly the quantity a description controls, and it is invisible to any instrumentation that starts at the tool.

The tracker had supported updated:>2026-03-01 and created:<2026-01-15 throughout. Nothing in the description said otherwise, nothing in the schema forbade it, and the model simply produced the shape it had been shown, then answered the user from the 25 most recently updated issues that matched the non-date part of the query. Those answers were fluent, specific, and about the wrong set of issues. There is no parse_error to grep for, no 4xx in the metrics, and no user complaint, because a plausible list of issues looks exactly like a correct one to the person who asked.

The middle row is the one that argues for choosing examples to span the capability rather than to be minimal. Two demonstrations cost 38 tokens here and covered both the common shape and the one the model was otherwise blind to:

{
  "description": "Search the issue tracker. Terms are implicitly ANDed; returns id, title, state, assignee and updated_at. Examples: query=\"status:open assignee:me\" (open issues for one person); query=\"project:api state:closed updated:>2026-03-01\" (closed in a date window)."
}

Over the next six weeks, on comparable traffic, calls carrying a date term went from 11 in 1,190 to 302 in 1,240, and the labelled accuracy of answers to time-bounded requests went from 48% to 89%. The lesson generalises past dates: an example is a statement about which part of your tool is reachable, so the parts you never demonstrate are, in practice, features the agent does not have.

One further check on that shift is worth running, against the demand rather than against the old figure, because it is the one that says whether the fix is the right size rather than merely in the right direction. Three hundred and two calls out of twelve hundred and forty is close to a quarter, and the hand-labelled share of requests that were time-bounded was a little over a fifth. The two agree within a few points, which is what you want to see. A pair of demonstrations that had produced date filters on half of all calls would be overcorrecting into requests that never asked for one, and a pair producing them on a twentieth would still be leaving most of the demand unserved. Matching the shapes you demonstrate to the distribution of real requests is the design rule, and this is how it gets checked after the fact.

Treating the example as code that is not under test#

The comparison people reach for is the stale docstring, and it is close enough to be misleading. A docstring that describes an old API misleads a human who is reading it, and humans are unreliable in a specific way: they skim, they cross-check against the implementation, they notice when the example does not match the function signature two lines below. The failure is slow and partial. A stale description does not mislead a reader; it changes the bytes the system emits, on every request, deterministically, from the moment the backend changes. There is no partial adoption and no gradual discovery. It is closer to a hardcoded constant in a hot path than to a comment.

Descriptions are consumed by a machine that acts on them immediately, which is what puts them on the code side of the line usually separating code from documentation. Everything a team already does with code is available for them — versioning, tests, review of changes, an alert when something breaks — and almost nothing a team does with prose is adequate, because the review practices for prose all assume a reader who can notice that something looks off.

Which suggests the treatment. If the examples in your tool descriptions are executable claims about the backend, execute them. The concrete version is a test that pulls the examples out of the tool definitions and puts them through the real validator:

import re
import pytest

from app.tools import TOOL_DEFINITIONS       # the same list sent as `tools`
from tracker.query import parse as parse_query

EXAMPLE_RE = re.compile(r'query="([^"]+)"')


def iter_query_examples():
    for tool in TOOL_DEFINITIONS:
        for match in EXAMPLE_RE.finditer(tool["description"]):
            yield tool["name"], match.group(1)


@pytest.mark.parametrize("tool_name,example", list(iter_query_examples()))
def test_description_examples_parse(tool_name, example):
    """Every query shown in a tool description must parse today."""
    parse_query(example)

Eleven lines, and it would have failed in the same continuous-integration (CI) run that shipped the new parser. The regex is deliberately dumb; a stricter version reads examples from a structured field rather than scraping prose, which is worth doing once you have more than a handful. The point is not the extraction mechanism but the assertion: an example in a description is a promise about production behaviour, and promises about production behaviour belong in the test suite.

A stronger version removes the opportunity for drift instead of detecting it after the fact, by making the description and the code that already exercises the query language read the same constant:

# app/tools/examples.py
QUERY_EXAMPLES: dict[str, list[str]] = {
    "search_issues": [
        "status:open assignee:me",
        "project:api state:closed updated:>2026-03-01",
    ],
}

# app/tools/definitions.py
from app.tools.examples import QUERY_EXAMPLES


def _demo(name: str) -> str:
    return "; ".join(f'query="{q}"' for q in QUERY_EXAMPLES[name])


SEARCH_ISSUES = {
    "name": "search_issues",
    "description": (
        "Search the issue tracker. Terms are implicitly ANDed; returns id, "
        "title, state, assignee and updated_at. "
        f"Examples: {_demo('search_issues')}."
    ),
}

The constant is now imported by the parser test above and, more usefully, by the tracker client's own integration suite, which runs the same strings against a live tracker in CI. When the parser changes, the examples fail in whatever job already covers the query language, in the same pull request that changed it, with a stack trace pointing at a file an engineer recognises as code. The description has stopped being prose that happens to contain a claim about the backend and has become a rendering of a claim the test suite owns. This costs one indirection and is worth it for any argument format that has a validator, which is most of the cases where an example was needed in the first place.

The same reasoning covers examples that are not queries. If the description shows an id, that id should be one your resolver can look up or an obviously-synthetic pattern the backend rejects loudly rather than silently. If it shows a date range, the format should be the one your parser accepts. If it shows an enum value, that value should still be in the enum — an example demonstrating status="pending" after pending was renamed to awaiting_review produces a call that fails validation on every request, and the schema's enum list will not save you, because the model reads the demonstration first and the enum is downstream of a decision already made.

When an example earns its place#

None of this is an argument for removing examples. The version of search_issues without one was worse than the version with a stale one in a specific way: the stale example produced a uniform, immediately visible 91% failure rate that took an afternoon to diagnose once someone looked, whereas the no-example version produced a scatter of invented dialects with a long tail of queries that parsed but meant something other than what the user asked. A tool wrapping any non-obvious argument format wants a demonstration, and removing demonstrations to avoid staleness trades a loud failure for a quiet one.

It is also not a claim that every description needs an example. If the arguments are a customer id and an ISO date, the schema and a sentence carry all the information there is, and an example adds tokens you will pay for on every turn without changing behaviour. Examples earn their place where the argument has a shape the model would otherwise have to guess — a query dialect, a structured path, a template, an ordering convention. Chapter 6 comes back to this when it works out which parts of a long description are load-bearing and which are decoration; examples land on both sides of that line depending on the tool.

The property that makes an example worth adding and the property that makes it dangerous are one property: it fixes the shape of the output more tightly than any sentence can. That is why it belongs in the test suite rather than in the review checklist, and why what it does not show matters as much as what it does — an undemonstrated capability is one the agent effectively lacks.

Everything so far has been about one tool at a time: how it is chosen, and what it demonstrates once chosen. That framing runs out as soon as two tools could plausibly answer the same request. Both descriptions can be accurate, current, and well-exampled, and the model still has to pick one — and when it picks wrong, there is nothing in either string that is false.