Streaming tool calls

Read these first

• Tool calling
• Streaming

What you will know

• Assemble tool-call arguments from fragmented streaming deltas

  Splits can occur inside quoted strings, escape sequences, or nested objects, so concatenation must preserve order before parsing.

• Handle prose and a tool call arriving in the same turn

  Users may see a natural-language preamble before execution, while the eventual call must remain separate from the text renderer and preserve event order.

• Prove that dispatching a partial tool call is unsafe by feeding an incomplete JSON argument and observing the resulting parse or validation failure

  A syntactically incomplete fragment can look plausible yet omit required fields or alter escaped content, so completion is a correctness boundary.

• Build a UI that shows a tool call forming

  The display should update incrementally without exposing malformed JSON or implying execution has started, and must remain usable when chunks arrive unevenly.

• Recover when a tool-call stream is cut off

  Treat timeout, disconnect, malformed finish markers, and cancellation as non-success; retain enough state to log and retry safely without duplicate side effects.

Chapters

1. 1

   Argument deltas and text deltas

   A streamed turn delivers a tool call's arguments as a dozen or more fragments of a JSON document. A text delta is meaningful at every prefix; an argument delta belongs to one specific call and means nothing until that call reports itself finished, so the only operation on it is append — no parse, no decode, no trim. Everything looks fine in a test with one short call per turn, which is exactly why the fixtures that matter use two calls and long arguments.

   Read chapter
2. 2

   Keying the argument accumulator

   A buffer keyed by tool name works until one turn emits two calls to search_orders with different filters, at which point two independent JSON documents interleave into a string that either fails to parse or yields one call carrying fields from both. Key by the per-call index the stream puts on every fragment, keep the stable call id from the announcement, hold one accumulator per stream, and concatenate fragments as opaque strings in arrival order.

   Read chapter
3. 3

   Rendering blocks in event order

   A renderer whose only rule is to append text deltas produces one continuous paragraph in which a preamble runs straight into the sentences built on a tool result, so a call that really executed leaves no trace and the reply reads as invention. The structure that fixes it is the one the provider is already sending: an ordered list of content blocks carrying their own coordinates, with text deltas mutating a text block and argument deltas going to the accumulator, and a call block that moves from preparing to ran. Stored in the same shape, the trace survives a reload, which is when anyone looks at it.

   Read chapter
4. 4

   Dispatching on partial arguments

   The gap between the moment a call's required fields first parse out of the buffer and the moment the call reports itself finished had a median of 210 milliseconds, and dispatching at the first moment looks like free latency. It is not: the remaining tokens can extend a value, resolve an escape differently, repeat a key, or add the optional field that scopes the query. Restricting it to read-only tools bounds the damage to reads and not to answers, because the speculative result is the one returned to the model. Speculative warming keeps most of the win and moves the dispatch decision back to completion.

   Read chapter
5. 5

   Displaying a call as it forms

   Piping the argument buffer to the screen shows a data format nobody asked to see, mid-token, occasionally carrying a value nobody meant to display. The safe version decodes the buffer with a tolerant parser on each delta and renders only the field values that have stabilised, in schema order, from a field map where anything unlisted defaults to hidden. Nothing on screen may claim the call is running until the dispatcher has actually run it, because a forming call may never be dispatched at all.

   Read chapter
6. 6

   Finalising a streamed tool call

   The terminal event says only that the provider has no more argument fragments for this call, which is a statement about the stream and not about the content. Three non-success shapes follow from that — no completion event at all, a completion event over a buffer that does not parse, and arguments that parse and fail the schema — and only the last can be answered in-band with an error result. The other two end the turn, and because a preamble and a 'preparing' indicator are already on the user's screen, each has to resolve into a stated failure rather than an indicator that never moves.

   Read chapter
7. 7

   Retrying a cut-off stream

   A stream error after the completion event lands in a window where the tool has already been dispatched, so re-issuing the turn produces a second identical call the model cannot know is a repeat — 89% of stream failures land before dispatch and the remaining 11% are the ones that pay twice. Deleting the retry removes one of many routes to a replayed turn and leaves the dispatcher with no memory. The guarantee belongs there instead, as a key derived from the conversation, the turn index, the tool and the canonical arguments, claimed in a ledger before the tool runs and forwarded downstream.

   Read chapter
8. 8

   Persisting an interrupted turn

   An aborted turn leaves you holding rendered prose and a call that never completed. The buffer is a string where the provider's schema wants an object, and a fabricated call needs a fabricated result beside it or the next request is rejected — which writes a record of a tool that never ran into every later turn and every replay. Persisting the prose alone is correct about the tool and silent about the interruption, so the follow-up turn carries a promise with no evidence: on a hundred replays of that history the model invented a ticket number 68 times, and none at all once a one-line interruption note was in the input.

   Read chapter