Three ways to ask for JSON

A language model answers in text, which is the format a person reads and the wrong format for everything waiting downstream. Take a document-extraction service: it receives a photograph of a paper invoice, runs it through OCR — optical character recognition, the software that turns a picture of text into characters — and hands the resulting text to a language model along with an instruction about what to pull out of it. The ledger on the far side needs a decimal it can add up, the reconciliation job needs a date it can compare, and neither of them can do anything with a sentence saying the invoice is for one thousand two hundred and forty dollars. Something has to turn the model's answer into fields.

Structured output is the name for having a model produce those fields directly, in a data format your code can load, so that no human and no fragile string-handling sits between the model and the database. The format is almost always JSON — JavaScript Object Notation, the text of braces, quoted keys and values that most web APIs already speak — because every language ships a parser for it and every model has read enormous quantities of it in training. The problem it exists to solve is a boundary problem: a program has to accept, from a system whose job is to generate plausible text, something it can treat as data without anyone checking it first. You meet it wherever a model's answer feeds a program rather than a person — a document-extraction endpoint, a classifier that has to return one of five labels a switch can read, a moderation service returning a decision and a reason code, or an agent choosing which function to call, since the arguments to that call arrive as a JSON object the runtime hands straight to your code.

In all of those the same question comes up, which is how much of the shape of that object you are actually guaranteed and how much you are hoping for. Providers answer it at three different strengths, and the three are easy to confuse because two of them travel on the same request parameter. The first is a plain instruction: the prompt ends with Return a single JSON object with these fields and nothing else, and the request carries no special parameters. The second turns on the provider's JSON mode, a flag that makes the provider steer generation so the reply is syntactically valid JSON. The third hands the provider a JSON Schema — a document, itself written in JSON, that declares which keys must appear and what type each one holds — and asks it to decode against it. The second and third are usually set on a parameter called response_format, which is why three quite different contracts look nearly identical in code.

What each of the three requests actually says#

The plain version is the one almost everything starts as, because it requires no provider support at all and works identically across vendors:

resp = client.chat.completions.create(
    model="gpt-4.1",
    messages=[
        {"role": "system", "content": EXTRACTION_INSTRUCTIONS},
        {"role": "user", "content": ocr_text},
    ],
    max_tokens=1500,
)
raw = resp.choices[0].message.content

The second adds one parameter. Providers name it differently, and most of them require the word "json" to appear somewhere in the prompt before they will accept the flag at all, which is a hint about how the feature works underneath:

resp = client.chat.completions.create(
    model="gpt-4.1",
    messages=[...],
    response_format={"type": "json_object"},
    max_tokens=1500,
)

The third replaces the flag with a schema, and asks the provider to enforce it during decoding:

INVOICE_SCHEMA = {
    "type": "object",
    "additionalProperties": False,
    "required": ["invoice_number", "issue_date", "total",
                 "currency", "status", "line_items"],
    "properties": {
        "invoice_number": {"type": "string"},
        "issue_date": {"type": "string"},
        "total": {"type": "number"},
        "currency": {"type": "string", "enum": ["USD", "EUR", "GBP"]},
        "status": {"type": "string", "enum": ["paid", "unpaid", "partial"]},
        "line_items": {
            "type": "array",
            "items": {
                "type": "object",
                "additionalProperties": False,
                "required": ["description", "quantity", "unit_price"],
                "properties": {
                    "description": {"type": "string"},
                    "quantity": {"type": "number"},
                    "unit_price": {"type": "number"},
                },
            },
        },
    },
}

resp = client.chat.completions.create(
    model="gpt-4.1",
    messages=[...],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "invoice", "strict": True,
                        "schema": INVOICE_SCHEMA},
    },
    max_tokens=1500,
)

Four extra lines of code separate the first request from the last, and the contracts they establish have almost nothing in common.

Counting the failures, class by class#

The way to see the difference is to send the same prompt body over the same documents in all three configurations, changing nothing else — same model, same temperature, same max_tokens, the ceiling on how much output the model is allowed to produce before it is cut off mid-sentence — and count what broke. Here is that run over five hundred scanned supplier invoices that had already been through OCR, against one prompt asking for six fields: invoice_number, issue_date, total, currency, status, and a line_items array. The classes are counted independently: a response that failed to parse was not also examined for missing keys, and a response that parsed was checked against the schema by hand rather than by the provider.

The forty-seven parse failures in the first column decompose into three causes, and they are worth naming because the next chapter is entirely about them: thirty-one responses wrapped the object in a markdown code fence — the triple-backtick delimiters a model puts around code so that a chat client renders it as a block — eleven prefixed it with a sentence of preamble along the lines of Here is the extracted invoice data:, and five appended a closing remark after the object. In none of those cases was the JSON itself malformed. The bytes the model produced contained a perfectly good object; the bytes surrounding it were the problem.

Those three causes add to the forty-seven exactly, and checking that they do is worth the few seconds. It establishes that the parse column has no residue in it: no fourth cause hiding under the total, no genuinely malformed JSON, nothing that a more capable model would have fixed by being better at JSON. All forty-seven are decisions about what to put around the object, which is a different kind of problem from failing to build one, and which is why the fix for them lives in the code that reads the response rather than in the prompt that asked for it.

JSON mode removes that entire column and removes nothing else. Look across the row for total: thirty-nine failures became forty-one, which is noise on a sample of this size rather than an improvement or a regression. The model was returning "1,240.00" as a string — comma as a thousands separator, exactly the way the number appears on the invoice image — and a string is a legal JSON value, so a mode that guarantees legal JSON has no reason to object. Same for status, where the model returned "overdue" fourteen times because the invoices said "OVERDUE" and nothing in the mode knows that your enum — the closed list of values a field is permitted to take — has three members and that is not one of them.

Calling a two-count difference noise is a claim worth checking rather than asserting, and the check is one square root. A count of events over a fixed number of trials wobbles from run to run by roughly the square root of the count itself, and the square root of forty is between six and seven. Two configurations behaving identically would therefore routinely produce counts differing by half a dozen, so a difference of two is comfortably inside what "no change at all" looks like. The same arithmetic is what makes the first row a real result rather than a lucky run: forty-seven going to zero is not a wobble any square root can manufacture.

A team that turns on JSON mode watches json.loads failures drop to zero overnight and sees no movement at all in support tickets about wrong invoice totals, and both facts are consequences of the same guarantee. JSON mode buys you bytes that parse. Which keys appear, what type each one holds, and whether an enum value is one you defined are all still decisions the model makes, so a well-formed object carrying total as the string "1,240.00" sails through the parser and lands in the ledger, where whatever coerces it to a number decides what happens next.

The requirement that the word "json" appear in the prompt is worth cashing in here, because what it discloses is what the flag does not do. A mechanism that made a model decide to produce a document would have no need of the prompt's cooperation. What this one governs is the form the output takes, and it leaves untouched the question of what the model was going to say. Ask for a paragraph of explanation and switch the flag on, and the provider is left steering a paragraph's worth of intent into brace-and-quote syntax, which is worse than either thing you might have wanted. So the vendor requires the intent to be stated in the prompt as well, and the two have to agree. That the feature needs both is the first evidence that a guarantee about bytes and a guarantee about content are separate purchases.

The belief that forms here is worth writing in the words somebody would actually use, because it is the sentence a reader composes confidently after reading a release note: we turned on JSON mode, so our JSON is valid now. Every word of that is true. What makes it dangerous is the step nobody takes next, which is to notice that valid JSON was never the property the ledger needed. The ledger needs a number where a number was promised. A parser that succeeds is evidence about syntax, the counter that just went to zero was a syntax counter, and the tickets keep arriving because nobody ever built the other one.

Why the third column is different in kind#

The first two configurations are the model following instructions with varying degrees of encouragement. The third changes the decoding process. At each step the sampler produces a probability distribution over the whole vocabulary, meaning every token the model is capable of emitting, and constrained decoding masks that distribution so that only tokens which can continue a string conforming to the schema retain any probability mass. Once {"invoice_number": "INV- has been emitted, tokens that would open a new object are unreachable; once the decoder is inside the currency value, the only continuations that survive the mask are the ones spelling USD, EUR or GBP. Provider implementations compile the schema into a grammar and evaluate that grammar token by token, which is why the first request carrying a new schema pays a one-off preprocessing cost of a second or two while the grammar is compiled, and subsequent requests with the same schema do not.

The consequence is that four rows of the table go to zero and stay there, because the outputs that would have violated the schema are no longer in the reachable set. The model's instruction-following has not improved at all. That is a much stronger guarantee than the previous two, and it is worth being precise about its edges, because the strength is exactly what makes people over-trust it.

Constrained decoding constrains tokens. Everything that is not a property of the token sequence remains untouched, and the bottom three rows of the table are the evidence. Six responses in the third column stopped at max_tokens, and a response that stops mid-object is a response the grammar was perfectly happy with right up to the point where generation ran out of budget — the schema says what a complete document looks like, not that generation will reach the end of one. Two responses were refusals: the model declined the task and returned an explanation, which most providers surface either as a separate refusal field on the message or, on the ones that do not, as prose where you expected a document. And nine of a hundred and twenty audited documents carried a total that was the right type, inside the right object, and factually wrong — the model had picked up the subtotal before tax on multi-page invoices where the grand total appeared on a later page.

Those three are not a list to memorise, because one property of the mechanism generates all of them. The mask acts on the choice of the next token and on nothing else, so anything expressible as "which token may come next" is inside its reach and anything that is not is outside it. Run the three through that test. max_tokens is a decision taken outside the grammar entirely: it stops generation whatever the mask would have permitted next, which is why what arrives is a legal prefix of a legal document that never reached its closing brace, and why the grammar has nothing to say about it — a grammar constrains what may follow, not whether anything will. A refusal is a decision about what to say rather than how to spell it, and no arrangement of schema-legal tokens spells I will not do this, so it has to surface somewhere the schema does not govern. Correctness is a relationship between the document and the world, and the world is not in the token distribution: the tokens spelling a subtotal and the tokens spelling a grand total are equally legal, equally typed, and equally decimal. Apply that same test to the next guarantee a vendor announces and the table below stops being something to look up.

Where the schema stops being the schema you sent#

There is a fourth survivor that does not show up in a single-provider experiment. JSON Schema is a large specification and providers implement subsets of it. In OpenAI's strict mode every property must be listed in required and additionalProperties must be false, which means an optional field has to be expressed as a nullable union rather than by omission from required. Constraints like minimum, maxLength and pattern are supported unevenly across vendors and versions; some reject the schema outright, and some accept it and silently do not enforce that keyword. A schema your SDK serialised cleanly and the API accepted with a 200 is not, by itself, evidence that every constraint in it is being applied to the decoder.

The last row of the guarantee grid is the reason the first two columns are not obsolete. Schema-constrained decoding is not available on every model you might want to route to, the schema dialect differs between vendors, and a fallback path that drops to JSON mode or to a plain instruction when the primary provider is unavailable is a normal thing to have in production. Any system with a fallback owns all three columns of that grid at once, and owns the parse failures that come with the leftmost one.

The validator is also doing a job that has nothing to do with distrust of the provider. It is the thing that converts a bad response into a counted bad response. Deleting it does not make the failures go away; it makes them arrive further downstream, in the ledger, as a support ticket, with the raw response long since discarded.

Each mode buys a different, narrow guarantee, and knowing exactly where each one stops is what tells you how much validation you still owe on your own side. A plain instruction owes you everything. JSON mode owes you everything except the parse. Schema-constrained decoding owes you truncation, refusal, and correctness, which is three of the six rows in the grid above and the three that cost the most to find late.

The forty-seven parse failures are where the next chapter starts. None of them were malformed JSON — every one contained a complete, valid object with something else wrapped around it — and the obvious fix, taking everything from the first { to the last }, is a rule that works on all forty-seven and fails in ways that are much harder to see than a json.loads exception.