Finalising a streamed tool call

Every rule in this topic so far waits for the call to report itself finished. The accumulator stops appending, the renderer moves the block out of preparing, and the dispatcher parses the buffer and runs the tool. That single event is being treated as the moment everything becomes true at once, and it is not: it says the provider has no more argument fragments for this call, which is a statement about the stream rather than about the content.

The provider can stop sending for reasons that have nothing to do with the model having written a complete document — the generation hit a token ceiling, a safety filter cut the output, its own upstream connection dropped and it closed the block on the way out. And a stream can end without emitting the event at all, in which case nothing fires, no exception is raised, and the code after the loop runs normally against an accumulator that is missing a call.

What the terminal event certifies#

content_block_stop, response.function_call_arguments.done and finish_reason: "tool_calls" all say the same narrow thing. The three non-success shapes that follow are distinct in cause, in what you can recover, and in what the user should be told.

The rightmost column is the only one that stays inside the loop from topic 5.8. A call that completed and parsed but whose arguments do not satisfy the schema is a well-formed call with a bad payload, and it has an id, which means you can answer it the way you would answer any other call: with a tool_result carrying is_error and a message describing what was wrong. The model reads that, corrects the arguments, and emits a second call. On a ticketing assistant this happened on roughly 0.3% of calls, almost all of them an enum the model had guessed — priority: "P1" against a schema expecting low | normal | urgent — and the corrected retry succeeded 94% of the time, at a cost of one extra round trip and about 900 milliseconds. The user never learns that any of it happened, and that is the right outcome.

The in-band correction — the model repairing its own arguments inside the same conversation, without the turn ever failing or the user being shown anything — needs a bound, because the model can fail the same schema twice for the same reason. Allowing two corrective round trips and then failing the turn is the shape most teams settle on; the second attempt on that assistant recovered a further 4% of the initially-invalid calls and the third recovered 0.1%, at which point you are spending two seconds and three inference calls to arrive at the same failure message you could have shown at the start. What makes the bound necessary rather than tidy is that a schema the model cannot satisfy — a required field describing something the user never provided — produces an identical failure on every attempt, so an unbounded loop is a loop.

Put the three recovery figures in a column and the choice of bound stops being a matter of habit. The first corrective round trip recovers 94 in every hundred invalid calls. The second recovers a further four, taking the running total to ninety-eight. The third recovers a tenth of one. Each attempt costs the same round trip as the one before it — about 900 milliseconds — so the second attempt buys four points for that price and the third buys a fortieth of a point for the same price. Two extra attempts is where the curve has flattened, which is what the "two seconds and three inference calls" is measuring: two further round trips at 900 milliseconds each is a little under two seconds, spent to arrive at a failure that was already the likeliest outcome once the second attempt had failed.

The shape of that curve is not an accident, and knowing why tells you when the bound should be lower. The calls the first attempt recovers are the ones where the model guessed a plausible value and the error message named the right one — P1 against an enum, a date in the wrong format. Once the model has been told the enum and still gets it wrong, the remaining population is dominated by calls where no correct value exists to be found: a required field describing something the conversation never contained. No number of attempts produces information that is not there, so the recovery rate does not decay gently towards zero. It falls off a cliff as soon as the recoverable population is exhausted.

The other two columns leave the loop, because there is no valid call to answer. A truncated buffer has an id and a name but no arguments you can trust, and a stream that ended before the completion event may not even have produced the announcement. Neither of those can be handed back to the model as a corrected turn; the only move is to fail the turn and decide separately whether to re-request it.

The lifecycle with its failure edges drawn#

The states a call passes through, and the transitions out of them, are worth having explicitly rather than implicitly:

Four of those edges leave the diagram without a dispatch, and the reason to draw them is that the code in the earlier chapters has only one of them. A switch on event type handles Accumulating → Finished and nothing else; every other exit is the absence of an event, and absences do not fire handlers. Abandoned in particular is reached by nothing happening, which is why it needs a timer rather than a branch.

Making absence into an event#

An idle timer between events is the mechanism, and it is a different thing from the request timeout most HTTP clients give you by default. A total-request timeout of 60 seconds does not help with a stream that died at t=3s, because the connection stays open and idle and the client waits out the remaining 57 seconds before raising. What you want is a deadline measured from the last event of any kind, reset on every event, sized above the largest legitimate inter-event gap. On a billing assistant the p99 gap between argument deltas was 84 milliseconds and the worst observed provider hiccup was 1.2 seconds, so a 10-second idle timeout has two orders of magnitude of headroom over normal behaviour and still surfaces a dead connection about six times sooner than the request timeout would.

Both of those comparisons are divisions, and doing them is how you size the timer for your own traffic rather than copying this one. Ten seconds over 84 milliseconds is a little over a hundred, which is the two orders of magnitude: the timer sits a hundred times above the gap that a ninety-ninth-percentile normal delta produces, so normal behaviour cannot trip it. Ten seconds also leaves nearly an order of magnitude over the worst real hiccup observed, the 1.2 seconds, which is the more demanding of the two constraints and the one worth sizing against. At the other end, ten seconds into a 60-second request timeout divides to six, which is the factor by which the user's wait shortens.

The general form is a timer above the largest gap you believe is legitimate and far below the point at which a person gives up. Those two bounds leave a wide range, which is fortunate, because the measurement that fixes the lower bound — the worst legitimate gap — is the one you have least data about. Sizing from the median gap rather than from the tail is the mistake to avoid: a timer set at a few hundred milliseconds would sit above the median by a comfortable-looking margin and below the ordinary provider hiccup, and would therefore abandon healthy turns at a rate proportional to how often that hiccup occurs.

The finalisation step then has one job, which is to turn every one of those exits into an explicit outcome rather than a fall-through:

from dataclasses import dataclass

@dataclass
class Dispatchable:
    call_id: str
    name: str
    args: dict

@dataclass
class Failed:
    reason: str          # "abandoned" | "unparseable" | "truncated"
    call_id: str | None
    name: str | None
    raw: str             # the buffer as it stood, for the log

Outcome = Dispatchable | Failed | SchemaError

def finalize(acc: Accumulator, stop_reason: str | None) -> list[Outcome]:
    out: list[Outcome] = []
    for buf in acc.live_buffers():          # announced, never stopped
        out.append(Failed("abandoned", buf.call_id, buf.name, buf.args))
    for buf in acc.completed:
        if stop_reason == "max_tokens":
            out.append(Failed("truncated", buf.call_id, buf.name, buf.args))
            continue
        try:
            args = json.loads(buf.args)
        except json.JSONDecodeError as e:
            out.append(Failed("unparseable", buf.call_id, buf.name, buf.args))
            continue
        errors = list(validator_for(buf.name).iter_errors(args))
        out.append(Dispatchable(buf.call_id, buf.name, args) if not errors
                   else SchemaError(buf.call_id, buf.name, args, errors))
    return out

Two details in that function are the ones people leave out. It iterates the accumulator's still-live buffers first, so a call that was announced and never stopped becomes a Failed outcome rather than quietly disappearing when the map is garbage collected — the invariant from chapter 2, that announcements must equal completions plus failures, is enforced here. And it consults the message-level stop_reason before trusting a completion event at all, because max_tokens is the case where the provider emits a perfectly well-formed block stop on top of a document the model never got to finish.

That case deserves a concrete look, because it is the one that reaches production. A create_ticket call whose description ran long against a 1024-token output ceiling produced a completion event over this buffer:

{"project": "BILLING", "title": "March invoice run failed",
 "priority": "urgent", "description": "The 03-14 run aborted after
  1,204 of 3,880 invoices. Affected accounts are listed in the
  attached export; the failure appears to originate in the tax

There is no closing quote, no closing brace, and json.loads refuses it, which is the outcome you want. A tolerant parser does not refuse it — it closes the string and the object and hands back an object whose four fields all satisfy the schema, and a system that reached for tolerant parsing at the dispatch boundary would create a ticket whose description ends mid-sentence in the middle of the most important clause. The display path in chapter 5 uses a tolerant parse precisely because nothing it produces is acted on; the dispatch path uses a strict one for the same reason in reverse.

Follow the tolerant parser's reasoning on that exact buffer, because the danger is in how reasonable each of its steps is. It reaches the end of the input inside an open string, so it supplies a closing quote — the only character that could legally appear there. It is then inside an open object, so it supplies a closing brace, again the only legal option. What comes back has project, title, priority and description, which is every field the schema requires, all of them of the right type. There is no step at which the parser did anything arbitrary, and there is no property of the resulting object that a validator could object to. The document is complete, well typed and schema-valid, and the description stops in the middle of the sentence naming the cause of the outage.

That is why the strict parser's refusal is the useful behaviour rather than the inconvenient one. Strictness is not a higher standard of correctness here; it is the only remaining place in the pipeline where the difference between what the model wrote and what the model was going to write is still visible. One character later, that information is gone — and everything downstream, including the schema validator, is examining a document that no longer records having been cut short.

Failing in front of a user who was already told something#

Validating a non-streamed tool call is a private matter. The response arrives whole, you parse it, it fails, you retry or you fail the request, and nothing has reached the screen. Under streaming, by the time the completion event is due the user has read a preamble in which the model said what it was about to do, and has spent a second or more watching a block labelled "Preparing create_ticket" fill in with a project and a title. Two commitments are already in front of them, and neither of them can be withdrawn by doing nothing.

The consequence is that every non-success has to move the call block into a terminal, stated state. An indicator that stops updating is read as an app that is still working; the user's model of the interface has no category for "this element will never change again."

The reason that reading is the natural one is worth naming rather than treating as user error. An indicator that is animating has been saying, on every frame, that something is in progress. Nothing about it changes when the underlying work stops, because the animation is driven by your renderer and not by the stream. So the last statement the interface made is still on screen, still moving, and still asserting the same thing it asserted while it was true. The user is believing the most recent claim available to them. Getting the block into a terminal state is not about being helpful; it is about stopping the interface from continuing to assert something that has ceased to be the case.

What the block should say differs by outcome, and the honest versions are short:

  ⊘ create_ticket — not run (connection lost)
  ⊘ create_ticket — not run (the request was cut short)
  ⊘ create_ticket — cancelled

Each of those states three things: which tool, that it did not run, and a cause specific enough to tell a user whether retrying is worth their time. None of them shows the buffer, the parse error or the schema violation, all of which belong in the log line beside the raw string. And each of them contradicts the preamble sitting above it, which is the point — the user has been told "I'll create that ticket for you now" and needs to be told, in the same view, that it did not happen. Leaving the preamble as the last word on the turn is the failure mode the whole chapter is guarding against.

What each outcome is worth writing down#

Behind the screen, each of those outcomes is worth one structured log line carrying the call id, the tool name, the byte length of the buffer, the number of fragments received, the failure reason and the raw string itself, truncated at a few kilobytes. The raw string is the part people omit on privacy grounds and then need during the incident, and a middle position — store it, redact the fields the display map already marks as sensitive, expire it in seven days — keeps both properties.

The metric worth alarming on is the ratio of abandoned and unparseable outcomes to announced calls, because it moves for reasons outside your code: a provider degradation, a proxy with a shorter idle timeout than yours, a new tool whose arguments routinely exceed the output ceiling. On the assistants described here that ratio sat between 0.5% and 0.9% of tool-using turns in steady state, which is a small enough number to be invisible in aggregate latency dashboards and a large enough one to generate several support tickets a week at moderate volume.

The reason a ratio is the right thing to alarm on rather than a count is that the denominator moves for reasons that have nothing to do with health. Traffic doubles on a Monday and a count of abandoned calls doubles with it while nothing has changed; traffic halves during an unrelated outage and the count falls while the failure rate is climbing. Dividing by announced calls removes the volume term from both directions, leaving a figure that only moves when the proportion moves. Announced calls is the right denominator specifically — not turns, and not requests — because every one of these outcomes begins with an announcement, so the ratio is failures over opportunities to fail.

Cancellation belongs in this list even though nothing went wrong. A user pressing stop, a client navigating away, a request-scoped context being cancelled by a deploy — each of these ends the stream mid-arguments, and each produces exactly the Abandoned state that a disconnect does. Treating cancellation as a fourth category with its own softer handling is a common way to end up with a code path that skips the finalisation step, because the cancel felt intentional and therefore harmless.

Three claims, then, and the stream makes one of them. The call is finished, which the terminal event asserts; the buffer parses, which a strict parser decides; the arguments satisfy the schema, which a validator decides. Only the third failure can be handed back to the model, and only the first is reached by an event — which is why the absence of an event has to be given one.

None of the above tells you whether to re-issue the request. Failed("abandoned") says a call did not complete in the stream you were reading; it says nothing about whether the tool behind that call ran, because the events that reach your process and the side effects that happen in the world are not the same timeline. The forty-one-second create_ticket was safe to abandon, since nothing was ever dispatched. A send_payment in the same state is a different question, and answering it wrongly is how the same payment goes out twice.