Token accounting in a streamed response

stream: true is a statement about the provider's response writer, not about its scheduler. Without it, the server generates the whole message, serialises it, sets content-length, and writes one body. With it, the server writes headers immediately and flushes a small event each time it has text to send. The GPU work is identical, the queueing behind it is identical, and the tokens are identical. The only thing that moves is the timestamp on the first byte your process observes.

Why that is a certainty rather than an observation is worth spelling out, because it is the argument you will have to make out loud in a room. The price of a request is a rate multiplied by a count of tokens. The count of input tokens is fixed by the prompt, which the flag does not touch. The count of output tokens is fixed by how many times the decoder ran, which is fixed by how much the model chose to say, which the flag does not touch either. The meter is therefore attached to the decoder, and stream: true is a statement about the component that writes the decoder's output to a socket. Two components, one downstream of the other, and the flag is on the downstream one.

The belief that argument displaces is tempting for good reasons rather than lazy ones. In most systems, arranging to deliver work incrementally does reduce the total: a paginated query touches fewer rows, an incremental build recompiles less, a streamed file needs less memory to serve. Each of those got cheaper because streaming let something be skipped. Here nothing is skipped. Every prompt token is prefilled and every output token is decoded in both arrangements, and the only thing that changed is when the bytes left the building.

That has two consequences, and teams tend to meet them in the wrong order. The first is financial and is a negative result: streaming is not a cost lever, and proposing it as one is a promise that will be measured. The second is operational and produces a support ticket in nearly every team that logs usage: the token counts, which a non-streamed response carries in the same object as the text, are carried in a streamed response by a different event that arrives at the end and is easy to never read. An overnight batch job is the case where the accounting is all there is, since nobody is watching the tokens arrive.

What the flag actually toggles#

Take a job that summarises support tickets in bulk: about 4,200 input tokens against a 600-token summary per ticket. At $3 per million input tokens and $15 per million output tokens, one ticket costs $0.0126 of input and $0.009 of output. Every one of those tokens has to be processed whether the response is written to the socket in one piece or in four hundred pieces. Prefill runs over the full prompt either way, decode runs one pass per output token either way, and the meter counts tokens rather than flushes.

Those two per-ticket figures are worth deriving rather than accepting, because the same two multiplications are the ones you will run against a real invoice. A rate quoted per million tokens is a price per token with the division deferred, so $3 per million input tokens is three dollars spread over a million of them, and 4,200 of them is 4,200 millionths of that three dollars — a little over a hundredth of it, which is the $0.0126 above. The output side is the same operation at a different rate: 600 tokens is six ten-thousandths of a million, and six ten-thousandths of $15 is $0.009. Add the two and one ticket costs $0.0216, which is the figure the table repeats in both of its columns because neither multiplicand is something the flag can reach.

Two rows in that table are the whole reason streaming exists, and two of them are the reason the rest of this topic is difficult. The total time barely moves — the streamed figure is marginally higher because the server does slightly more work flushing four hundred times than once — and the connection is held for the same duration in both cases, which surprises people who imagine that a non-streamed call is somehow cheaper to hold. It is not: your process is blocked on that socket for nine seconds regardless.

The reason that row cannot move is the same reason the token rows cannot. A non-streamed call still requires the provider to decode every token before it can serialise anything, and the response it eventually writes cannot be written earlier than the last token exists. Your side has to stay open until that write arrives, because there is nowhere else for the answer to come from. Both arrangements therefore hold a socket open for the length of the generation, and the only difference is whether that socket is silent throughout or busy throughout. Turning the flag on does not shorten the window in which a load balancer, a proxy or a closing laptop lid can intervene. It fills the window with traffic.

The last two rows are the ones the remaining chapters are built on. Usage moves out of the response object and into an event, and a failure that happens after the headers have gone out can no longer be an HTTP status code, because the status code was written and spent at 340 milliseconds.

Where the token counts went#

The symptom is a usage column that is null for every streamed row and populated for every non-streamed one, sitting next to stored assistant text that is perfect. The cause is that delta events carry text and nothing else. Here is the accumulator that produces it, and it is the natural thing to write:

text_parts = []
usage = None

for chunk in client.chat.completions.create(
    model="gpt-4.1",
    messages=messages,
    stream=True,
):
    text_parts.append(chunk.choices[0].delta.content or "")
    usage = chunk.usage          # None on every delta chunk

store(text="".join(text_parts), usage=usage)

Every delta chunk looks like this, and its usage field is genuinely absent:

{
  "id": "chatcmpl-9x",
  "object": "chat.completion.chunk",
  "model": "gpt-4.1",
  "choices": [{"index": 0, "delta": {"content": " Lisbon"}, "finish_reason": null}],
  "usage": null
}

The loop assigns None on the first chunk and reassigns None four hundred more times. Nothing errors, the text is right, and the accounting is silently empty — which is the worst failure mode available, because it looks like a data pipeline problem rather than a client problem and gets investigated in the wrong place.

It is worth asking why the number is not simply on every delta, since the provider plainly knows how many tokens it has decoded so far. Two of the three quantities on a usage record are settled at different times. Input tokens are known as soon as prefill has run over the prompt, which happens before the first delta can exist, so nothing prevents a provider from reporting them early — and Anthropic does. Output tokens are not known until the decoder stops, because what is known at any moment mid-stream is a running count rather than a final one, and a running count is a different number that happens to look like the same one. A provider could put that running count on every event, at the cost of some bytes on every event and of an ambiguity about which of the four hundred figures is the answer. Both major providers chose instead to write it once, after the text.

Token counts are not free to produce mid-stream, so providers only emit them once the message is finished, and OpenAI makes you opt in:

stream = client.chat.completions.create(
    model="gpt-4.1",
    messages=messages,
    stream=True,
    stream_options={"include_usage": True},
)

text_parts = []
usage = None

for chunk in stream:
    if chunk.usage is not None:
        usage = chunk.usage       # final chunk: choices is []
        continue
    if chunk.choices:
        text_parts.append(chunk.choices[0].delta.content or "")

With include_usage set, the stream gains one extra chunk before [DONE] whose choices array is empty and whose usage is populated:

{
  "id": "chatcmpl-9x",
  "object": "chat.completion.chunk",
  "choices": [],
  "usage": {"prompt_tokens": 4207, "completion_tokens": 613, "total_tokens": 4820}
}

Note the guard on chunk.choices in the fixed loop. Code that reaches for chunk.choices[0] unconditionally worked fine until the day someone turned include_usage on, and then it threw an IndexError on the last chunk of every request — after all the text had been rendered, so the user saw a complete answer and the server logged an exception. That is a common sequel to fixing the null-usage bug.

Anthropic's stream splits the same information across two events instead of appending one. message_start carries input_tokens up front, since prefill has already happened by the time the first event is written, and message_delta near the end carries the cumulative output_tokens alongside stop_reason. Both events appear in the event trace shown in the previous chapter. The practical consequence is the same: the number is never on a text delta, and a client that only handles content_block_delta will accumulate a correct message with no idea what it cost.

Cumulative, total, and the figure that gets added twice#

Having found where the numbers live, there is a second way to get them wrong, and it appears when someone writes one accumulator meant to work against both providers. The natural shape is usage.output += chunk.usage.output_tokens, summed over every event that carries the field, and it is right on one provider and wrong on the other.

OpenAI's usage chunk is a total for the request and appears once, so summing is harmless. Anthropic's message_delta carries a running total of output tokens for the message so far, and a message with several content blocks — a text block, a tool use block, then more text — emits message_delta more than once. Summing running totals double-counts, and it double-counts more on the longer, more structured responses, which produces the reported symptom that "complex answers cost more than the model should be charging". They do cost more, which is why nobody questions the figure. Assignment rather than addition is the fix for a cumulative field, and it is worth a comment in the code saying which one you have, because the two lines look identical in review.

The size of the error follows from what a running total is, and a small case makes it visible. Suppose a message emits the field twice: once after the text block, reporting the tokens produced so far, and once at the end, reporting the tokens produced altogether. The second figure already contains the first, because that is what "so far" and "altogether" mean about the same message. A loop that adds them therefore records the whole message plus its own opening section, and the overcount is exactly the value of every intermediate report. Three blocks give two intermediate reports to add in, four blocks give three, and so on — which is why the error is not a fixed percentage but one that grows with how structured the answer was. A short prose reply emits the field once and comes out correct, so the bug never appears on the requests you would think to check it with.

Input accounting has a similar edge. Prompt caching, where the provider keeps the processed form of a prompt prefix so that a later request reusing that prefix skips the work of reprocessing it, splits the input side into buckets that bill at different rates — ordinary input tokens, cache writes at a premium, cache reads at a discount — and they arrive as separate fields. Code that reads input_tokens alone and multiplies by the standard rate will produce a number that is confidently wrong in whichever direction your cache hit rate points, and it will drift as the hit rate changes, so the error is not even a constant you could correct for.

The step being skipped there is that a rate is only a rate for tokens of one kind. Multiplying a count by a price assumes every item in the count carries that price, and prompt caching is precisely the arrangement under which they do not: a token that was served from a cached prefix cost the provider almost nothing to reprocess and is priced accordingly, while a token being written into the cache for the first time costs more than an ordinary one. The fields arrive separately because the prices are separate. Summing them into a single input_tokens and applying the standard rate is the same arithmetic error as averaging two products' unit prices and multiplying by how many you sold.

The counts you will not get#

There is a case where none of this saves you, and it is the case the cancellation chapter later in this topic is about. If the client aborts at token 200 of an 800-token answer, the stream ends before the usage-bearing event is written, so you have partial text and no counts at all. You were still billed. Any accounting system that treats missing usage as zero will under-report exactly the requests that cancellation was supposed to make cheaper, which means the metric you would use to prove cancellation works is the metric cancellation breaks. The workable answer is to record a locally computed estimate — a tokeniser count over the text you actually received — as a distinct field from the provider-reported figure, so that the two can be reconciled and a systematic gap between them is visible rather than absorbed.

The circularity in that is worth stating slowly, because a dashboard built on the wrong side of it will report the fix working while the fix does nothing. The usage event is written after the text, so a stream that ends early does not reach it. Ending early is what cancellation does. A system that treats an absent usage record as zero therefore assigns a cost of nothing to precisely the requests that cancellation was introduced to make cheaper, and it does so whether or not any cancellation signal ever left the process. The measurement improves in proportion to how many usage records were lost, which means it improves fastest when the mechanism is most broken. A separate estimate field is not belt and braces here. It is the only column in the table that is populated on the requests the question is about.

Even in the normal case, treat the provider's number as the authority and yours as a check. Local token counts drift from billed counts because of system prompt handling, tool schemas, and image blocks. Reconcile monthly against the invoice and treat a persistent gap as a bug rather than as rounding: a team logging 128.4M output tokens against a billed 124.9M is 2.8% high, which is small enough to live with and large enough to be worth explaining, while a gap of 40% is a summing error of exactly the cumulative kind above.

Both halves of that judgement come out of one subtraction and one division, and it is worth seeing which is which. The difference between the logged figure and the billed one is three and a half million tokens, and three and a half million against a billed hundred and twenty-five million is under three hundredths of it — small enough that it is plausibly drift in the local tokeniser, which segments text by its own rules and was never shown the system prompt, the tool schemas or the image blocks the provider also counts. A gap of 40% cannot be explained that way, because no plausible amount of tokeniser drift is two fifths of a month's traffic. The size of the gap is what tells you which kind of bug you have: a small one is measurement drift, and a large one is an arithmetic error of the cumulative kind.

The accounting rules that follow from all of this are short. Read usage from the event that carries it rather than from the deltas, opt in where opting in is required, assign a cumulative field rather than adding it, split the input side into its cache buckets before multiplying by a rate, and store a local estimate alongside the provider's figure so that the requests which end without a usage event are visible as estimates rather than as zeroes. None of it is difficult; all of it is invisible until somebody joins your logs to an invoice.

What none of it makes streaming yet is a reliability feature, though the premise of this topic is that it is one. So far it is a user-experience feature with an accounting wrinkle. It becomes a reliability feature the moment you notice that a nine-second held connection is nine seconds in which a load balancer, a CDN, a serverless platform, or a user closing a laptop can intervene — and every one of those interventions has to be expressible in a protocol that has already sent its status code. Before any of that, there is an argument to settle that every team has when the streaming ticket is written, because someone always proposes doing it over a WebSocket instead.