Truncated JSON and lenient parsers

Whether a truncated response is caught for free or written to a database as an ordinary record is decided by one thing: what sits between the API response and the parse. Nothing about the model, the request or the failure differs between those two outcomes.

This chapter is about that layer. It starts with the default behaviour, which is better than people expect — strict JSON parsing catches a mid-generation cut with probability approaching one, because the grammar has the global structure that prose lacks. It then catalogues the four common setups that dismantle the property, all of them added for good reasons by someone who was not thinking about truncation, and ends with an experiment that tells you which case your own code is in rather than leaving you to assume.

The default behaviour is the loud one#

Strict JSON parsing is unforgiving about balance, and that unforgivingness is doing real work. A JSON object is not valid until its final } arrives, a string is not valid until its closing quote arrives, and an array is not valid until its ] arrives. Cut a generation at a uniformly random position inside a JSON object and you land inside an unterminated string, an unterminated object, or between a key and its value — all of which are hard syntax errors.

raw = '{"id": "A-1", "total": 4200, "notes": "Vendor disp'
json.loads(raw)          # raises JSONDecodeError

How reliably it fails depends on the shape of the object, and the odds are good. Take a typical extraction response of 240 characters across five fields: the positions from which a cut yields something a strict parser accepts are essentially none, because the closing brace is the last character and every position before it leaves an object open. Even a cut that lands neatly between two complete key-value pairs produces {"id": "A-1", "total": 4200, — a trailing comma and no closing brace, both fatal. Strict JSON is one of the few output formats where truncation is caught with probability approaching one, and that is a property worth defending rather than one to route around.

The reason the odds are that good is structural, and stating it precisely makes it portable to formats this chapter does not list. An opening brace incurs an obligation that only its matching brace discharges, and so does every array bracket and every string quote inside it. Nesting means those obligations are settled in reverse order of opening, so the outermost one — the object's own closing brace — is settled by the very last character of the document. Stop anywhere before that character and at least one obligation is still outstanding, whichever position you stopped at. For a document whose top level is an object or an array, it follows that no proper prefix of it is a valid document: the only prefix that parses is the whole thing. That is the exact reverse of the property prose has, where every prefix is as well-formed as the whole, and it is why the same cut that hides perfectly inside a summary field cannot hide inside a JSON one.

Other formats are far weaker. A truncated CSV loses its last row, which is indistinguishable from a document that had one fewer row. Truncated Markdown is as silent as prose. A truncated fenced code block loses its closing fence, which is loud if you extract content by matching fences and silent if you strip the first fence and take the rest. YAML sits in between: cutting mid-value often still parses, because YAML infers structure from indentation rather than requiring terminators. If you have a choice about the format your model emits and you care about catching truncation, strict JSON is the format that catches it for you.

Ranked that way the formats stop being a list to remember. What decides whether a format catches a cut is how much of its correctness can only be established at the end. CSV keeps its structure inside each row, so a row that never arrives violates nothing and the file is merely shorter than it would have been. YAML keeps its structure in indentation, which is local in the same way, so a cut usually leaves a smaller document that is still a legal one. JSON hangs a single outstanding obligation on the final character of the file. Any format you are considering can be placed on that scale before you test it, by asking what would have to be true for the document to prove itself finished, and whether the answer mentions the end.

Four ways the structure gets closed for you#

Partial-JSON parsing exists to solve a genuine problem. If you are streaming a structured response into a UI and want fields to appear as they arrive, you need a parser that accepts {"id": "A-1", "tot and hands you {"id": "A-1"} so the id can render before the total exists. Every library in this space is designed to make an unfinished object usable, which is exactly right during a stream and exactly wrong once the stream has ended. Teams that build the streaming UI first and the batch job second often reuse the parser, inheriting a behaviour that was a feature in one context and a hazard in the other.

The thing that flips between those two contexts is time rather than data. While a stream is open, an unfinished object is the expected state of the world; every object is unfinished for most of its life, and a parser that objected to it would be objecting to normal operation. Once the stream has closed, an unfinished object is a fact about how the generation ended. The bytes are identical in the two cases and the correct response to them is opposite, which is why the choice of parser cannot be made from the bytes and has to be made from where in the lifecycle the call sits. A partial parser used on a completed response is a component still answering the question it was built for, at a moment when that question has stopped being the right one.

Repair passes get added after a bad week. Models sometimes wrap JSON in a markdown fence, sometimes append a sentence of commentary, sometimes emit a trailing comma. Somebody writes a helper that strips fences, finds the outermost braces, and balances what is inside; error rates drop; everyone moves on. The helper never advertises that it now converts truncation into a plausible object, because the person who wrote it was not thinking about truncation.

Constrained decoding — structured outputs, grammar-constrained sampling, whatever your provider calls it — masks the sampler at each step so that only tokens which keep the output conforming to your schema can be selected. The guarantee it offers is real and valuable: when generation completes, the output validates. What it does not offer is a guarantee about generation completing. Depending on the implementation, an interrupted constrained generation gives you either a valid prefix, which a strict parser still rejects and so leaves you in the loud case, or a structure the serving layer closed off for you, which parses cleanly and is missing content. Both providers document the caveat explicitly in their structured-output guides — Anthropic's notes that if stop_reason is max_tokens the output may be incomplete, and OpenAI's says the same about a length finish reason — and the reason they bother is that the shape guarantee is so strong that callers stop asking whether the content is finished.

Being precise about what the masking can reach explains why that caveat has to exist rather than being a gap somebody could close. The constraint is applied at each sampling step. It works by tracking where the grammar currently stands — after a key, inside a string value, expecting either a comma or a closing brace — and driving the probability of every token that could not legally come next to zero. That machinery decides which token may follow. It has no way to insist that another token must follow, because "there is more to come" is not a property of any individual candidate token; whether the object is finished is a property of the document as a whole, and the document as a whole does not exist until generation has ended. The schema guarantee is therefore a statement about generations that completed. Halt the loop early and the grammar state machine is caught mid-parse, holding a legal prefix of a legal document — which, by the paragraph above, is exactly the thing that is not a document. Whether you then receive that prefix, or a version the serving layer has closed off on your behalf, is a decision made by the stack rather than by the schema, and it decides whether you are in the loud case or the silent one.

The word "guaranteed" is doing the damage. A feature that promises schema-conforming output is understood, entirely reasonably, as a feature that removes a class of worry, and the class it removes is malformed output rather than unfinished output. Those are different failures with different causes, and conflating them is easier when the vendor documentation leads with a guarantee and puts the truncation caveat in a note further down.

Finding out which case your own code is in#

Do not take on faith which case you are in. Force a truncation and watch what your own parser does with it.

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=24,                     # deliberately far too small
    system="Extract id, total, and notes as JSON.",
    messages=[{"role": "user", "content": invoice_text}],
)

print(response.stop_reason)            # max_tokens
print(repr(response.content[0].text))  # '{"id": "A-1", "total": 4200, "not'

record = parse_model_json(response.content[0].text)   # your own parser
print(record)

There are only two possible outcomes and both are informative. If the call raises, your parser is strict and truncation is loud in your code today — which is worth writing down, because the next person to add a repair helper will change that without knowing it. If it returns a dictionary, you have just watched a truncated response become an ordinary-looking record, and everything downstream of that line is now working with data it believes is complete.

Run the experiment at three or four cap values rather than one, because the position of the cut changes what you learn. A cap of 8 tokens cuts before the first field completes and will break almost any parser. A cap around the middle of the object cuts inside a string value. A cap one or two tokens short of the natural length cuts just before the closing brace, which is the case a repair pass handles most convincingly and therefore the case most likely to slip through unnoticed. It is also the most realistic, since production truncations cluster near the natural length rather than being uniformly distributed — the responses that get cut are the ones that were slightly longer than usual, so they were nearly done.

That clustering follows from which responses get cut at all, and the reasoning is short enough to do here. A response is truncated only if the answer it wanted to write was longer than the cap. Among the answers that qualify, the ones that overshoot by a little are far more numerous than the ones that overshoot by a lot, because answer lengths pile up around a typical value and thin out as they get longer. So conditioning on truncation selects mostly for answers that were just over the line, and an answer just over the line is one whose cut lands close to its own end. The practical consequence is that the truncations you will actually meet in production are the ones with the most fields already complete and the least missing — which is to say, the ones a repair pass reconstructs most convincingly, and the ones a reviewer reading a sample is least likely to question.

Before running anything, a two-minute audit narrows down what you expect to see. Grep the dependency manifest for json-repair, partial-json, dirty-json, demjson and their equivalents in your language. Grep the codebase for any hand-rolled helper whose name contains repair, sanitize, extract_json or clean sitting near a model call. Check whether the request sets output_config.format, response_format, or whatever your provider calls constrained decoding. Any hit means you are probably in the silent case, and the experiment is confirming a suspicion rather than exploring blind.

Ordering the repair after the check#

The instinct after this exercise is to rip out the repair pass. That is usually wrong, because the repair pass is still doing the job it was added for, and removing it reintroduces the fence-and-commentary failures that motivated it.

What is wrong is where it sits. A repair pass is a recovery mechanism, and recovery mechanisms must not run before the check that decides whether recovery is warranted. Repairing a response that ended at end_turn and happened to carry a trailing comma is correct. Repairing a response that ended at max_tokens is fabricating data — the closing brace it inserts was never generated by anything, and the object it produces asserts that notes ends at "Vendor disp", which is a claim no component in the system is entitled to make.

It is worth looking at what a repair pass has to invent in order to return that object, because the inventions are all of one kind. It supplies a closing quote for notes, and in supplying it asserts that the value ends where the bytes ran out. It supplies the closing brace, and in supplying that asserts no further keys were coming. Neither assertion is derived from anything the model produced, and neither could be: a routine whose contract is to hand back something parseable has no other move available to it. What comes out is well-typed, schema-valid and internally consistent, and every field in it is as trustworthy as it appears except the last, which is a fragment wearing the shape of a complete value. Downstream, nothing can separate a note that genuinely reads Vendor disp from one that was cut there, because by the time the row is read both are strings and strings carry no history.

def parse_model_json(response):
    if response.stop_reason != "end_turn":
        raise IncompleteGeneration(response.stop_reason)   # check first
    return repair_and_load(response.content[0].text)       # recover second

The same ordering rule applies to retries. A generic "retry on parse failure" wrapper around an extraction call will, on a truncated response, re-run the identical request with the identical cap and truncate again, burning three attempts and then either failing or falling through to whatever the last-resort path is. Retries help with transient errors. Truncation is deterministic given the same inputs, so retrying it without changing max_tokens is a loop that cannot succeed.

Strict parsing, then, is a free truncation detector that most pipelines eventually give away, usually in exchange for something they also needed. The exchange is fine as long as somebody knows it happened: keep the repair, keep the partial parser for the stream, keep the schema guarantee — and put the finish reason check in front of all of them, where it decides whether the bytes are worth interpreting at all.

There is one case none of this covers, and it is the one that produces the longest-lived incidents. Suppose you have no repair pass, no partial parser and no auto-closing; suppose your parser is strict and your code is in the loud case. There is still a position at which a generation can be cut that leaves behind syntactically perfect, fully balanced JSON containing every field your schema demands — and the object is wrong anyway. Not repaired, not patched, not reconstructed by any library. Genuinely valid, genuinely complete-looking, genuinely incomplete.