Dispatching on partial arguments

The proposal this chapter is about is a good one on the whiteboard. A tool call's required fields tend to appear early in the argument object, so there is a window, after the last one is readable and before the call reports itself finished, during which the tool could already be running. Instrumenting a streaming assistant's accumulator across ten thousand production turns put the median of that window at 210 milliseconds and its p90 at 390, on a p95 tool-using turn of 4.8 seconds against a product target of under four. Limiting the trick to read-only tools appears to close off the risk, since nothing can be written twice.

The window is real and the safety argument does not hold, for a reason that has nothing to do with how good your parser is.

What those milliseconds are#

The gap is the model still writing. An argument object of 610 characters is around 155 output tokens, and at the 110 tokens per second this deployment was getting, the whole object takes about 1.4 seconds to generate. The required fields tend to appear early, because schema order and the model's own habits put identifiers before descriptions, so by 400 milliseconds in you often have every field the tool declares as required and the model is still producing optional ones — a notes string, a filter object, a closing brace. Stream transit and event dispatch account for single-digit milliseconds of the measured gap. Essentially all of it is generation.

Every figure in that paragraph comes from two conversions, and doing them once makes the whole chapter's arithmetic reproducible on your own deployment. Characters convert to tokens by division: 610 characters over 155 tokens is a little under four characters per token, which is the ordinary ratio for English text with punctuation in it. Tokens convert to seconds by division as well: 155 tokens at 110 tokens per second is a little over 1.4 seconds, which is how long the model spends writing that object. The 400 milliseconds at which the required fields are typically all present is the same conversion run backwards — four-tenths of a second at 110 tokens per second is somewhere around forty-odd tokens, and forty-odd tokens is enough for an opening brace, a project key and a title.

Knowing which term dominates is what tells you the gap is not yours to optimise. The measured window is the time between the last required field appearing and the call finishing, and both of those events are produced by the model's writing speed. Transit and event handling contribute single-digit milliseconds, so they are two orders of magnitude below the quantity being measured and cannot move it. A faster network, a faster event loop and a faster parser would each leave the 210 milliseconds where it is.

That matters because it tells you what you are actually buying. You are not removing overhead from your own stack, and you are not overlapping work that was serialised for no reason. You are starting the tool before the model has finished saying what it wants, and the only way that is safe is if the remaining tokens cannot change the request. They can.

Four ways the finished document differs from the readable one#

The first three are cases where a value you already read is not final. A tolerant parser handed {"customer": "Acme" will close the open string and hand you Acme, which is a complete, plausible, schema-valid company name, and there is nothing in the buffer that distinguishes it from a name the model had finished writing. The escape case is worse in a quiet way: the fragment ends between the backslash and the u, and a repair parser resolves the dangling escape to something — usually a literal backslash, sometimes by dropping it — so the query string you dispatch differs from the model's in a character that no log line will draw attention to.

The fourth case is the one that survives every objection about parser quality, because no parser is at fault. The required field was present, complete, and correct. What arrived afterwards was an optional field that narrowed the query, and a search dispatched without it is a correct query against the wrong universe. Optional in the schema means the model may omit it; it does not mean the model's request means the same thing without it.

Proving it against a recorded stream#

This is cheap to demonstrate, and worth doing once on your own traffic rather than taking on faith. Record one real turn's argument deltas, then evaluate every prefix of the reassembled buffer as though you had dispatched there:

import json
from jsonschema import Draft202012Validator

fragments = json.load(open("create_ticket_deltas.json"))  # 47 strings
schema = json.load(open("create_ticket_schema.json"))
validator = Draft202012Validator(schema)
final = json.loads("".join(fragments))

buf, parsed, valid, wrong = "", 0, 0, 0
for i, frag in enumerate(fragments, start=1):
    buf += frag
    try:
        candidate = json.loads(buf)      # strict, not tolerant
    except json.JSONDecodeError:
        continue
    parsed += 1
    if validator.is_valid(candidate):
        valid += 1
        if candidate != final:
            wrong += 1
            print(f"prefix {i}/{len(fragments)} would dispatch: {candidate}")

print(f"parsed={parsed} valid={valid} valid_but_wrong={wrong}")

On the create_ticket capture this prints parsed=3 valid=1 valid_but_wrong=1, and the offending line is prefix 31 of 47:

prefix 31/47 would dispatch: {'project': 'BILLING', 'title': 'March invoice run failed'}

Three of the 47 prefixes were syntactically complete JSON, which happens whenever a split lands on a closing brace of a nested object or on the document's own brace. One of those three satisfied the schema, because project and title were the only required fields. It was also wrong: the finished call carried a priority of urgent and an assignee, neither of which had been written yet. A strict parser gave that result. A tolerant parser, which closes open strings and braces, produces a valid-looking candidate at far more prefixes — on the same capture, 29 of 47 — and every one of the extra 26 is a guess.

The difference between three and 29 is the difference between reading and inventing, and it is worth being exact about it. A strict parser accepts a prefix only when the model happened to stop on a boundary that closes every structure it opened, which is why three of the 47 prefixes qualify: it is a coincidence of where the splits fell. A tolerant parser accepts far more because it manufactures the missing closers itself — a quotation mark to end an open string, braces to close open objects. Each closer it adds is a decision the model had not made yet. Subtracting the strict count from the tolerant one leaves twenty-six prefixes that are documents the parser wrote rather than documents the model wrote, and none of them is marked as such when it reaches your dispatcher.

Repeat the experiment with the required-field rule the latency proposal actually uses, dispatching at the first prefix where a tolerant parse contains project and title, and it fires at prefix 9, 1.1 seconds before the call completed and with four of the six eventual fields still unwritten.

Check that 1.1 seconds against the generation figure, because the two are the same measurement from opposite ends. The whole object takes about 1.4 seconds to write and arrives as 47 fragments, so a fragment is a little under thirty milliseconds of writing on average. Prefix 9 is therefore reached somewhere around a quarter of a second in, and the remainder — the 1.4 seconds less that quarter — is the 1.1. What that says plainly is that the required-field rule does not dispatch a fraction of a second early. It dispatches when the model is about one-fifth of the way through saying what it wants, and the four unwritten fields are four-sixths of the request.

Read-only does not contain the blast radius#

The safety argument is that a premature read costs a wasted query, and a wasted query costs a few milliseconds of database time. That accounting stops at the database, and the failure does not.

Follow the search_orders case through. The dispatcher fires on {"customer": "Acme"} and gets back 47 rows for a different company. The call completes half a second later with {"customer": "Acme Holdings GmbH", "status": "refunded"}, and the code that returns tool results has one result to return, tagged with this call's id, and it is the speculative one. The model reads 47 rows of orders as the answer to a question it did not ask, and having no way to detect the substitution, it writes a summary: "I found 47 orders for Acme, of which none are marked refunded." The user reads a confident, specific, entirely wrong answer. Nothing in the system logged an error, because at no point did anything fail — the query was valid, the rows were real, the schema passed, the model responded. The only artefact is a log line showing a search_orders call whose arguments do not match the arguments the model emitted, and nobody reads that log line until a customer escalates.

The step that turns a wasted query into a wrong answer is worth isolating, because it is the one the safety argument never gets to. The protocol from topic 5.8 requires exactly one tool_result per call, tagged with that call's identifier. Your dispatcher has fired twice but the conversation has room for one answer, so something has to choose which result to send, and the speculative one is the only one that exists when the code reaches that point. From there the model is not comparing the result against the arguments it wrote; it has no mechanism for doing so and no reason to suspect a mismatch, because a tool_result is by construction the answer to the call it names. So the wrong rows enter the conversation with full authority. Every subsequent sentence in the turn is reasoning correctly from them.

That is the general form of the failure, and it explains why "read-only" is the wrong axis to reason on. The classification read-only versus write describes what the tool does to your systems. The damage here is done to the conversation, and every tool writes to the conversation — that is what returning a result means. A taxonomy that sorts tools by their effect on the database has no column for the effect the topic actually cares about.

A dispatch on unfinished arguments is therefore not a wasted query with an occasional retry. It is an unlogged substitution of the model's request with a truncated version of itself, and the model's next turn is built on it. Read-only bounds the damage to reads; it does not bound the damage to answers, and answers are what the product ships.

Inside the safety argument there is a narrower claim that is also false, which is that a read tool has no side effects. Reads cost money and quota, and a speculative dispatch on every call roughly doubles both: the order search here was backed by a vendor API billed at $0.004 per query, so 180,000 tool calls a month moves from about $720 to about $1,440, and the extra spend buys nothing on the 29% of turns where the speculation turns out to be wrong. Reads also consume rate limit from the same bucket the real query needs, so the speculative call and the real one can race for the last permit in the window and the real one is the one that loses. And a good number of read endpoints write something on the way past: a get_customer that appends to an access audit log, a document search that updates a "recently viewed" list, a CRM read that stamps last_accessed on the record. Those logs get read by auditors and by support agents, and "this record was accessed 14 times in one minute" is a sentence about your speculation rather than about anyone's behaviour.

The money in that paragraph is worth walking, both for the figure and for the unit. A per-query price multiplied by a query count gives a cost over whatever period the count covers, so $0.004 a query at 180,000 queries a month is a monthly figure — four thousandths of a dollar taken 180,000 times, which is the $720. Speculating on every call issues a second query for every first one, so the count doubles and the cost doubles with it, to $1,440. The increment is therefore $720 a month, and that is what buys the milliseconds. Carrying that number into an annual budget means multiplying by twelve rather than quoting it as it stands, which is the mistake that makes this look cheap.

The version to ship#

Speculative warming keeps the latency win and moves the dispatch boundary back where it belongs. On a read-only tool, when the buffer's tolerant parse first satisfies the schema, issue the query as a cache fill — into a request-scoped cache keyed by a hash of the exact argument object used — and do not surface, log, or return that result to anything. When the call reports itself finished, parse and validate as normal, hash the real arguments, and if the hash matches the speculative one, serve the cached rows; otherwise discard the cache entry and dispatch properly. The dispatch decision still happens exactly once, on completion, against arguments the model finished writing.

What that costs is measurable. On the order-search traffic the speculative arguments hash-matched the final ones 71% of the time, so the expected saving is roughly 0.71 × 210ms ≈ 150 milliseconds at the median rather than the full gap, and the read volume against the orders index rises by close to 100% because every call issues a query whether or not it is used. Whether 150 milliseconds is worth doubling read QPS is a question with a real answer at your traffic level, and it is a different question from the one the original proposal asked.

The multiplication in that saving is an expectation, and expectations are easy to misread as guarantees. On the 71% of turns where the speculative arguments hash-matched, the cached rows are already there and the full 210 milliseconds is saved. On the remaining 29% the cache entry is discarded and the query runs from scratch on completion, so the saving on those turns is zero — and the doubled read has already been paid for. Multiplying the rate by the per-turn saving is what averages those two outcomes into the roughly 150 milliseconds quoted, which means no individual turn ever saves 150 milliseconds. Turns save 210 or nothing. That distinction matters if what you are trying to move is a p95 rather than a median, because a p95 is decided by the slow tail and the slow tail is disproportionately made of turns where the model changed its mind late — which is precisely the population where the hash does not match.

Cheaper levers on the same turn#

Set against the other things that shorten the same turn, 150 milliseconds is a small number:

Prompt caching earns its row because tool definitions are large and are re-sent on every request in a multi-turn loop — eleven tools with full JSON Schema ran 3,100 tokens on this assistant — and having the provider keep that unchanging prefix of the request and reuse it across calls, which is what prompt caching is, shortens the stretch before the model writes anything, which the user is waiting through just as much as the argument generation.

Two cheaper levers usually beat it. Shortening the argument schema removes generation time directly — dropping two rarely-used optional fields from create_ticket took the median argument object from 155 to 96 tokens, which is around 540 milliseconds at 110 tokens per second, more than twice what the speculation was chasing. That figure is the same two-step conversion as before, run on the difference: 155 tokens less 96 is fifty-nine tokens no longer being written, and fifty-nine tokens at 110 per second is a little over half a second. Nothing is being raced and nothing is being cached — the tokens simply do not exist, so the time to produce them does not either. This is the one lever in the table that reduces the work rather than overlapping it, which is why it is larger than everything above it and why it is the one with a real cost in the last column: the fields are gone, and with them whatever they let the model express. And Anthropic's fine-grained tool streaming, which stops the provider buffering and validating arguments before sending them, moves fragments to you sooner without moving the completion event; it is worth knowing about here mainly because it makes malformed accumulated JSON more likely to reach your code rather than less, which the next chapter's display path and the one after that's validation path both have to survive.

One further variant comes up in every version of this discussion: dispatch speculatively, then re-dispatch on completion if the arguments changed, discarding the first result. That is safe with respect to correctness and buys nothing on latency in the case that matters, because whenever the speculation was wrong you pay the full call anyway, plus the wasted one. Its only value is when the speculation was right, which is warming with extra steps.

The rule underneath all of it is that a buffer's readability is not a statement about the model's intent. A prefix that parses is a prefix that parses, and the tokens still to come are exactly the ones that scope, qualify and correct what came before. Dispatch on completion, warm a cache if the milliseconds are worth it, and spend the effort on the levers that shorten generation rather than on the one that races it.

The buffer, then, is readable long before it is actionable, and the gap between those two states is hundreds of milliseconds during which the user is looking at something. Showing them the call taking shape is a genuinely good idea, provided the thing on screen never claims more than the stream has said.