The constrained decoding guarantee

Constrained decoding — sold under names like structured outputs or strict mode — is the provider feature that compiles a JSON schema into a constraint on generation, so that a response cannot violate the schema rather than merely being asked not to. It is a genuine improvement over prompting a model to "respond in JSON" and hoping, and it removes an entire genus of parsing bug. It also arrives with a corollary that sounds obvious and is wrong in an expensive way: that the assertions in your eval suite which check the schema are now verifying something the provider has already made impossible. Careful teammates reach that corollary independently and for good reasons, which is why it usually arrives as a pull request rather than as an argument.

This chapter separates three properties that get collapsed into one. What the constraint enforces, which is narrow and structural. What it does not enforce, which is most of what your suite has ever caught. And which code paths it covers at all, which is fewer than the number of paths a response can take between the provider and your assertion. Getting these apart is what decides whether a set of schema assertions is dead code or the only thing standing between a malformed object and a database write.

What the constraint does at decode time#

The mechanism is worth knowing because the limits follow from it directly. The schema is compiled once into a state machine over the token vocabulary; at every decoding step the sampler is handed a mask that sets the logit of any token which cannot continue a valid document to negative infinity — the logit being the raw score the model assigns each candidate next token before those scores are turned into probabilities — and sampling happens over what remains. If the automaton is inside a string value for a key typed "integer", no token containing a quote character is reachable. If four of five required keys have been emitted, the closing brace is unreachable until the fifth is.

Two steps in that description carry the whole guarantee, and both are worth going slowly over. The first is why a schema can be turned into a machine at all. A schema fixes the key names, fixes the type of each value, and in strict mode fixes that no other keys may appear, so at any point inside a half-written document the set of characters that could legally come next follows from where you are in the document and nothing else. After a key name and its closing quote, the only legal next character is a colon. Inside a value declared as an integer, the legal continuations are digits. With one required key still missing, a closing brace is not among them. A model emits tokens rather than characters, and a token is a fixed run of characters, so a token is reachable exactly when the whole run it spells is a legal continuation. Compiling the schema means working that question out once for each position the machine can be in, against every entry in the vocabulary, and keeping the answers.

The second step is what the mask does to the sampling, and it is the reason this guarantee is a different kind of assurance from the others in this topic. The scores the model produces are turned into probabilities by exponentiating them and dividing each by their total, and exponentiating negative infinity gives zero — not a very small number, zero. A candidate at zero probability occupies none of the range the sampler draws from, so no draw can land on it: not at a higher temperature, not under a different seed, not with a more insistent prompt. Invalid tokens are not made unlikely. They are absent from the set being chosen among, at every step, and a document assembled one step at a time out of legal continuations cannot arrive illegal.

Setting that beside the arrangement it replaces is what shows how unusual it is. A retry loop lets the model produce whatever it produces, parses the result, and asks again when the parse fails. Its residual failure rate is the per-attempt failure rate multiplied by itself once per attempt, which falls quickly and never reaches zero, and each attempt costs a whole generation of latency and tokens. Some number of attempts has to be the last one, and whatever that number is, there is a rate of malformed responses beyond it that still reaches your code. Constrained decoding does not push that rate lower. It deletes the branch: on the path the schema was sent down, there is no malformed document for a retry to catch, because at no step was one available to be produced.

The masking argument is a structural one, and it delivers exactly the properties that are structural. Every key in required appears. Every value has the declared JSON type. Every enum-typed field holds one of the listed members, because the alternatives were the only reachable continuations. Nothing appears outside the object — no Here's the JSON you asked for:, no fenced code block, no trailing commentary — which removes the regex that used to hunt for the first {.

There are costs, and they are worth knowing because they show up as latency and as surprising refusals. The automaton is compiled per schema and cached, so the first request against a new schema pays a one-off penalty that is usually tens to a few hundred milliseconds; subsequent requests do not. Providers accept only a subset of JSON Schema in strict mode — typically additionalProperties: false is mandatory, every property must be listed in required, and recursion depth and total property count are capped. And a grammar that forbids the token the model wanted does not make the model smarter; it makes it emit the best reachable token, which is occasionally a worse answer expressed in a valid shape.

Conformance and validity are different questions#

The properties above are all about form. The assertions that fired in a working suite over the last quarter were mostly about content, and no amount of decode-time masking speaks to content at all.

Why masking cannot reach content is settled by what the mask is a function of. At every step the mask depends on one thing, which is where the automaton has got to in the document. It does not depend on the customer, on the order table, on the refund policy, or on anything the prompt said, because none of that is part of the machine that was compiled. The mask can therefore delete a candidate for being structurally impossible and for no other reason at all, and among the candidates that survive, the model's preferences decide exactly as they would have decided with no schema in the request. Constraining is a projection of what the model was going to say onto the set of well-formed documents. It re-weights; it does not correct. Schema conformance is a property of the document on its own, whereas validity is a property of the relationship between the document and the world, and the decoder has access to only one of the two.

The two status rows are the projection argument in miniature and repay reading together. All four enum members are legal continuations at that point in the document, so the mask removes none of them, and the choice between them is therefore the unconstrained model's choice reached by the unconstrained route. What the guarantee did for that field was fix the alphabet the decision was made from, and it had nothing whatever to say about the decision. A document can be wrong in every value it holds and remain perfectly schema-valid, because shape and meaning are settled by different things at different moments — the shape by a machine that reads only the document so far, the meaning by a model under no obligation to be right.

The citations row is the one that catches teams out most often, because it looks like the schema covers it. Several providers accept a schema containing minItems, minLength, pattern and format, validate the schema itself, and then silently ignore those keywords when building the grammar — they constrain structure and type, not cardinality or lexical shape. So {"citations": []} conforms, {"eta_date": "not a date"} conforms against "format": "date", and {"order_number": "hello"} conforms against "pattern": "^ORD-\\d{6}$". The format and non-emptiness assertions of the previous chapter are therefore not redundant even on the happy path; they are checking precisely the keywords the decoder dropped on the floor.

Whether your provider honours a given keyword is a five-minute experiment and is worth running rather than assuming, because the failure mode is silent. Define a schema with "citations": {"type": "array", "items": {"type": "string"}, "minItems": 3} and a prompt that gives the model nothing to cite; run it fifty times at temperature

1. If minItems is enforced by the grammar you will get fifty arrays of at least three elements, some of them fabricated. If it is ignored you will get empty arrays, and you now know that the keyword is documentation for the model rather than a constraint on the decoder. The same probe works for pattern, minLength and format, and the answer differs by provider, by model, and occasionally by API version, which is why it is a probe rather than a fact to memorise.

Worth separating from all of this is the older, weaker mode usually called JSON mode, which guarantees only that the output parses as JSON and says nothing about your schema at all. The two are frequently confused in code that sets response_format without passing a schema, and the resulting object is valid JSON with whatever keys the model felt like. If you inherit a service and want to know which regime you are in, look for whether a schema is being sent and whether strictness is enabled; the difference in what you may delete from your suite is total.

The completion caveat is separate and sharper. The grammar guarantees that whatever has been emitted is a valid prefix of a valid document. It does not guarantee the document finishes, so a response that hits max_tokens mid-object comes back as {"order_number": "ORD-448192", "message": "Your order is in tr — perfectly conformant as a prefix, unparseable as JSON. The provider reports this in stop_reason, the field naming why generation ended, and an eval harness — the runner that executes each case, calls the model on its behalf and applies its checks — that reads only the body sees a parse error with no cause attached. This is the single most common way a strict-mode response fails to parse, and it is why a json.loads stays in your case checks regardless of what the provider promises.

The guarantee covers one path through your system#

Even where the constraint holds completely, it holds on the requests that carry the schema. Draw the routes a response can take on its way to an assertion and the coverage gaps become visible as edges without the guarantee on them:

Each of those edges is an ordinary production decision that somebody made for a good reason. The fallback exists because a provider incident should degrade the product rather than take it down, and the fallback is often a different model — sometimes a different vendor — whose strict-mode support differs or is off. The cheaper retry model exists because timeouts are expensive and a smaller model answers faster. The refusal path bypasses the schema by design; a safety decline is not an instance of your object type, and code that assumes it is will crash on the one response class you least want to crash on.

There is one more edge the diagram cannot show, because it exists inside the eval harness itself. Suites replay recorded responses — for speed, for determinism, for the frozen fixtures of chapter 7 — and a replayed response never touches a decoder at all. It is whatever bytes were on disk when someone saved them, including bytes saved before the schema tightened. A suite that trusts the provider guarantee is trusting a component that half its runs do not invoke.

Post-processing is the quietest of the five, and in most codebases it is where the schema stops being true. Between the provider's bytes and your assertion there is usually a function that trims whitespace, lowercases an identifier, coalesces a null into a default, merges a cached customer record, redacts an email address, or renames a key from the model's vocabulary to your domain's. Every one of those can produce an object the original schema would reject, and none of them ran inside the decoder.

def normalise(payload: dict) -> dict:
    payload["order_number"] = payload["order_number"].strip().upper()
    payload["citations"] = [c for c in payload["citations"] if c in known_docs]
    return payload

Two lines, both sensible, and the second can empty an array that the model populated. If your assertion runs on the model's raw output you never see it; if it runs on the normalised object — which is what the rest of your system will consume — you catch it the first time the filter is too aggressive. Assert on the object the caller receives, and keep the raw response beside it so a failure can be attributed to the model or to your own code.

What the assertions are, once they are not redundant#

The bet is best made explicit in the case — here with Pydantic, the Python library that turns a class declaration into a runtime validator and raises when an object does not match it:

from pydantic import BaseModel, Field, field_validator

class RefundDecision(BaseModel):
    order_id: str = Field(pattern=r"^ORD-\d{6}$")
    approved: bool
    refund_cents: int = Field(ge=0)
    reason_code: str
    citations: list[str] = Field(min_length=1)

    @field_validator("reason_code")
    @classmethod
    def known_code(cls, v: str) -> str:
        assert v in POLICY_CODES, f"unknown reason code: {v}"
        return v


def check_refund_case(raw: dict, case) -> None:
    decision = RefundDecision.model_validate(raw)     # form
    assert decision.order_id == case.order_id          # identity
    assert orders.exists(decision.order_id)            # referential
    assert decision.refund_cents <= case.order_total_cents
    assert decision.refund_cents <= POLICY_CAP_CENTS   # policy bound
    assert decision.approved == case.expected_approved

The first line duplicates the provider's guarantee on the primary path and is your only guarantee on the other four. The five that follow are claims the decoder could not have made and the schema could not have expressed: that the identifier matches this case, that it names a row in your database, that the amount is inside both the order total and the policy ceiling, and that the decision itself is the expected one. Notice how little of the value sits in the schema half. Once structure is free, the interesting assertions are the ones that reach outside the response.

There is a symmetrical error worth naming, because engineers who take this chapter seriously sometimes land on it. Asserting in an eval suite is not a substitute for validating at runtime. The suite tells you that the model behaved on 240 recorded cases; it does nothing about request 240,001 on a Sunday night. The same RefundDecision model belongs in the service, rejecting before any side effect, and the eval assertion exists to tell you when the model started producing things that validation would reject — early, in a pull request, rather than as a spike in a rejection counter.

Constrained decoding, runtime validation and eval assertions are three controls at three different points, and none of them subsumes another. The decoder makes a structural promise about one request path. Runtime validation rejects anything that reaches your side effects, whatever path it arrived by. The eval assertion attributes a change to a diff, before the diff merges, which is the only one of the three that can tell you a model upgrade started emitting empty citation arrays. Deleting the third because the first exists trades an early, attributed signal for a late, aggregate one.

Everything in these two chapters has read one thing: the response body. The assertions coming next cannot. Whether the agent called issue_refund with amount_cents as the integer 4200, how many prompt tokens a case consumed, and how long the provider spent generating are not properties of the answer text at all, and a harness that stores a string per case has already thrown them away.