Two protocols share the name streaming JSON

Streaming structured output is asking a model for a JSON object — a value with declared fields that your code will parse and act on, rather than prose a person reads — and receiving that object in pieces while the model is still writing it, over a connection that stays open for the duration. It exists because the alternative is dead time. An ordinary request/response call holds the whole answer back until generation finishes, which for an invoice object is three to four seconds of blank panel or idle worker, and the early fields are usable long before the last one exists.

Two delivery formats carry almost all of that traffic. Newline-delimited JSON (NDJSON) is what self-hosted servers such as Ollama speak: one complete object per line. Server-Sent Events (SSE) is the one-way HTTP stream whose messages are data: lines separated by blank lines, which nearly every hosted provider speaks and which topic 2.9, on the wire format and the buffered decoding loop it requires, covers in its own right. This chapter is a layer above that one: the transport is settled, and what remains is whether the unit it delivers is a value or a piece of one. You meet the question wherever a model's output is a shape rather than a paragraph — a panel filling itself in field by field, an extraction job feeding a queue, an agent reading a tool call's arguments as they arrive — and in each the client has to decide what one delivered unit is worth.

What one event is carrying#

In the first protocol, each unit on the wire is a whole JSON document that stands on its own. Ollama's NDJSON is the clearest example, and the same shape appears in log pipelines, in docker logs --follow, and in every JSON-Lines file you have ever jq'd. The stream is a sequence of documents; the newline is the delimiter; each document is meaningful the moment you have it.

{"model":"llama3.2","created_at":"2025-03-04T11:02:14.1Z","response":"{\"cust","done":false}
{"model":"llama3.2","created_at":"2025-03-04T11:02:14.2Z","response":"omer_","done":false}
{"model":"llama3.2","created_at":"2025-03-04T11:02:14.3Z","response":"name\":","done":false}
{"model":"llama3.2","created_at":"2025-03-04T11:02:16.9Z","response":"","done":true,"eval_count":184}

Look closely at what that trace actually contains. Every line is a complete object and parses cleanly, and the response field inside each one is a fragment of a different JSON document — the extraction result the model is writing. Both protocols are present at once, stacked. The outer one is object-per-event and the inner one is delta-based, and a sixty-line client that treats the outer layer as objects and the inner layer as opaque text to be concatenated is exactly right.

The second protocol carries fragments in the envelope's payload without any inner encoding to hide behind. Anthropic streams tool inputs as input_json_delta events whose partial_json field is a raw slice of the argument JSON:

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"tool_use","id":"toolu_01Qp","name":"record_invoice","input":{}}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"input_json_delta","partial_json":"{\"customer_"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"input_json_delta","partial_json":"name\": \"Mar"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"input_json_delta","partial_json":"tinez SA\", \"amount\": 12"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"input_json_delta","partial_json":"00}"}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

OpenAI's Responses API does the same thing under different names: response.function_call_arguments.delta events carry a delta string, and a terminal response.function_call_arguments.done carries the whole arguments string at once. The content_block_start event announces "input": {}, which is an empty object and not an empty result — it is the header of a value whose body has not been sent yet.

A loop written for the first protocol and pointed at the second calls JSON.parse on each data: line, succeeds, then reaches for a response field that does not exist here, finds nothing, and appends nothing. The loop that works reads partial_json and appends that. Rendering the accumulated buffer's last successful parse is the sibling mistake in the same family, and it is where a customer_name of "Mar" and an amount of 12 where the invoice said 1200 come from.

Both of those wrong values are recoverable from the trace above, and recovering them once turns a plausible story into a checkable one. Concatenate the partial_json fields in the order they arrive. After the first delta the buffer holds {"customer_. After the second it holds {"customer_name": "Mar. After the third it holds {"customer_name": "Martinez SA", "amount": 12. Only the fourth closes the document, and the amount in the closed document is 1200. Look at what the third prefix contains: 12 is a complete JSON number, and nothing in the bytes received so far marks it as the opening of a longer one, because a number ends at the delimiter that follows it and that delimiter is in the next event. So any client that produces a value out of a buffer which has not closed — by keeping whatever parsed most recently, or by closing the open containers to force a parse — has "Mar" and 12 available to it as perfectly legal readings of the bytes it holds. Neither is a corruption of the data. Both are prefixes being reported as values.

The framing guarantee is about the envelope#

A teammate proposing JSON.parse on every SSE data: line is invoking a real guarantee. SSE dispatches an event at the blank line that ends it, so by the time your parser hands you an event, the event is whole; a data: line is never delivered half-written, and if the connection dies mid-event, the incomplete event is discarded rather than dispatched. That framing property holds regardless of what the payload means.

The guarantee covers the envelope. It says nothing about whether the value you care about fits inside one envelope. When the provider is streaming one JSON document as token deltas, a complete envelope carries an incomplete value, and JSON.parse applied to the accumulated payload throws, while JSON.parse applied to the envelope returns a perfectly valid object containing a fragment. Both statements are true at the same time, which is why the argument between two engineers about this can run for twenty minutes with neither of them wrong.

Providers did not choose the delta protocol for tool arguments out of carelessness. The model produces tokens, and a token boundary has no relationship to a JSON boundary — "amount": 1200 may be emitted as "amount / ": / 12 / 00, and the server would have to buffer until the value closed in order to hand you whole fields. Some of them do buffer, which is why Anthropic's default behaviour chunks tool JSON into pieces that are usually short and complete-ish, and why the Fine-grained tool streaming documentation exists to turn that buffering off: with the beta header set, the deltas arrive with no server-side accumulation and no validation, so you get lower latency and an explicit promise that the JSON you receive may be incomplete or invalid at the end. The buffered default is a compromise between the two protocols, and it is a compromise rather than a solution, because a buffered chunk is still a fragment.

The claim that a token boundary has no relationship to a JSON boundary rests on how the vocabulary was built, and it is worth one level down because it is what makes the delta protocol unavoidable rather than merely convenient. A model does not emit characters. It emits tokens drawn from a fixed vocabulary, and that vocabulary was assembled by repeatedly merging character sequences that occur together often in training text. The merges are selected by frequency, not by meaning, so one token can carry a closing quote, a comma and the first letter of the next key, and a four-digit number can arrive as one token or as four. Nothing in that construction consulted JSON's grammar, and it could not have: the vocabulary was frozen long before anyone asked this model for an object. A server that wanted to hand you whole fields would therefore have to hold tokens back until it had parsed far enough to know a value had ended — which is buffering, which is the latency streaming was bought to remove. The two protocols travelling under one name are two answers to that trade, and the fine-grained mode is the one that refuses to make it on your behalf.

The third column is worth naming because it is where a client lands if it reads the body with for chunk in response.iter_content(4096) instead of splitting on newlines. Some endpoints stream a single JSON document over Transfer-Encoding: chunked with no event framing at all, and there the transport hands you whatever the network happened to deliver.

Reproducing the failure on purpose#

The reason this survives code review and dies in production is that the failure is a function of how the bytes were sliced, and dev slices differently from prod. A local server on loopback will frequently hand your client the entire response in one read. The same response over the public internet, through a load balancer, arrives in a dozen pieces. You can force the bad case deterministically without waiting for load:

import json

DOC = b'{"customer_name": "Martinez SA", "amount": 1200, "currency": "EUR"}\n'
STREAM = DOC + b'{"customer_name": "Okonkwo Ltd", "amount": 340, "currency": "GBP"}\n'

def slices(payload, size):
    for i in range(0, len(payload), size):
        yield payload[i:i + size]

def naive(chunks):
    """Parse every chunk, swallow failures, keep the last success."""
    last, attempts, wins = None, 0, 0
    for chunk in chunks:
        attempts += 1
        try:
            last = json.loads(chunk)
            wins += 1
        except ValueError:
            continue
    return last, attempts, wins

print(naive(slices(STREAM, 4096)))   # one chunk, and it still fails: two documents
print(naive(slices(STREAM, 48)))     # (None, 3, 0)

The 4096-byte case is instructive on its own: the whole payload arrives in one read and json.loads still raises, because two concatenated documents are not one document. Shrink the slice to 48 bytes and nothing parses at all, so last stays None and the caller receives an empty result with no error anywhere in the logs. Between those extremes is the case that produces mangled output — a slice size around 70 bytes, where each chunk happens to contain one complete line plus the beginning of the next, the complete line parses, the fragment is swallowed, and the record that straddled the boundary vanishes. Which records vanish depends on where the boundaries fall, which depends on payload sizes, TLS record sizes, and whatever your proxy does. That is why it is clean in dev and dirty at 300 requests per second.

Those three outcomes are a function of one variable — the slice size relative to the size of one record — and naming the three regimes is what turns an intermittent haunting into something predictable. When the slice is larger than the whole payload, every chunk holds more than one document, nothing parses, and you find out on the first request. When the slice is well under one record, no chunk holds a complete document, nothing parses, and you find out never: the swallowing except converts every failure into a continue, so the caller receives an empty result and no exception is raised anywhere. The regime that mangles rather than empties needs a client that does some framing of its own — one that splits on newlines inside a chunk, as the sixty-line client in the scenario below does — and it loses only the records that straddled a cut while every other record comes through intact. That is the output nobody investigates, because it is correct about most of what it returns and there is nothing to count.

The patch is a boundary-aware read rather than a smarter try/except. For NDJSON it is small enough to write inline:

def ndjson(chunks):
    buffer = ""
    for chunk in chunks:
        buffer += chunk.decode("utf-8")     # see chapter 2 on why this decode is wrong
        while "\n" in buffer:
            line, buffer = buffer.split("\n", 1)
            if line.strip():
                yield json.loads(line)
    if buffer.strip():
        yield json.loads(buffer)

The shape is the one you already apply to SSE when you split on the blank line: hold a buffer, cut at delimiters you find in it, keep the remainder for next time. What changes between protocols is the delimiter and what the cut piece means.

Three properties make that loop correct, and each is a line you can point at. buffer belongs to the function rather than to the iteration, so bytes that arrive without a delimiter behind them are still present when a later chunk brings one. The cut is a split with a limit of one, so everything after the first newline goes back into the buffer whole instead of being handled as though it were finished. And the trailing if buffer.strip() is there because a stream may end without a final delimiter, which is the one case the while loop cannot reach — leave it out and every response loses its last record, a defect that presents downstream as an off-by-one in something else entirely. Stated as a rule: never treat a unit as finished until the thing that finishes it has arrived, and keep everything after that thing.

Which protocol to emit when you own both ends#

The choice is forced on you when you are talking to a provider, and it is yours the moment you relay through your own endpoint. A relay can accept the provider's delta protocol and re-emit an object-per-event protocol of its own design, so that its clients never see a fragment. In practice that means sending one line-delimited JSON message per settled field:

{"seq":1,"field":"customer_name","value":"Martinez SA"}
{"seq":2,"field":"amount","value":1200}
{"seq":3,"type":"complete","fields":3}

Every line parses, every consumer is trivial, and browsers, mobile clients and downstream services all read it the same way. Spring Boot shops publish this pattern under the name NDJSON streaming, and the Spring AI: Streaming LLM Tokens with NDJSON write-up is a reasonable tour of the plumbing. The cost is that the work did not disappear — the relay is now the component that has to parse a half-finished document to decide when a field has settled, and it has to define what complete means and send it explicitly. What you buy is that the hard part exists once, in a service you deploy, rather than in every client you ship.

There is a real loss in the trade. Field-granular events mean a 400-word summary field arrives as one message at the end of its generation, so the user watching that field sees nothing for four seconds where the delta protocol would have given them words at 300 milliseconds. Relays that care about both usually run a hybrid: deltas for the long prose fields, settled values for everything else. OpenAI's own streaming guide describes the same split from the other side, with response.output_text.delta for prose and a separate .done event carrying the final value.

The layer choice does not settle the value layer#

Choosing the right layer to parse at does not tell you how to read a value before it finishes. If you are on the delta protocol and you want anything on screen before content_block_stop, you are going to parse a prefix of a JSON document, and a prefix is not valid JSON by construction — the ndjson helper above cheats by waiting for a delimiter that a half-written object does not have.

This also is not a transport question. Whether the bytes arrive over SSE, a WebSocket, or a chunked response changes the framing you get for free and changes nothing about the value layer; a WebSocket message is a complete frame carrying a fragment exactly as an SSE event is. And it is not solved by the provider SDK, which will happily hand you typed events and even an accumulating helper — right up until you relay the stream through your own endpoint, at which point you are the one defining the framing your client sees, and the two protocols are yours to keep straight.

The two protocols are a distinction about what one delivered unit is worth, and almost every silent corruption in a streaming client is a client applying one protocol's rule to the other's bytes. An object-per-event stream wants a parse per event and no accumulation. A delta stream wants accumulation and a parse that is deliberately not per event. There is no loop that is correct for both, and the cost of guessing is a service that returns an object for every request and is quietly wrong about a third of them.

There is one more thing the ndjson helper above gets wrong on purpose, marked in the comment. Decoding each chunk to text independently assumes chunk boundaries fall between characters, and the network makes no such promise. That assumption, and the three others hiding beside it, are what the next chapter is about.