Schema-constrained generation

Read these first

• Getting JSON out

What you will know

• Define a schema and have the provider constrain generation to it

  Include required fields, enums, nullability, and representative invalid cases so the generated contract is tested rather than merely declared.

• Reproduce the difference between prompt-only formatting and provider-enforced output on the same test cases

  Prompting can produce extra keys, missing fields, or fenced text that passes casual inspection but fails a real validator.

• Handle a provider that supports strict mode and one that does not

  Provider support may differ by model, schema keyword, and SDK path, so fallback behavior must be explicit rather than inferred from a successful request.

• Show that constrained output eliminates a whole class of parse errors

  Constrained decoding removes malformed-grammar errors, not all parse errors: hitting max_tokens still truncates valid JSON mid-object, which is why 1.8 and 5.5 exist.

• Measure output quality, latency, and token usage with and without constraints

  Constraints can narrow useful continuations or add decoding work, so compare semantic accuracy and tail latency instead of assuming syntactic validity is free.

Chapters

1. 1

   Constrained decoding versus prompting

   Same schema, two mechanisms: a system prompt that describes the shape, and a provider mode that restricts which tokens the decoder may emit. Run both over the same cases and one whole class — unbalanced braces, bare keys, a fence wrapped around the object — disappears from one of them. A teammate who ran fifty prompt-only responses through `json.loads` and saw fifty passes measured the wrong thing; the real validator surfaces the extra keys and missing required fields.

   Read chapter
2. 2

   Exporting a model to JSON Schema

   The class you parse results into and the JSON Schema you send the provider are the same declaration doing two different things at runtime. Types, required-ness, enums, nullability and a handful of keywords survive the export; custom field validators and cross-field rules do not, so a rule rejecting a past `due_date` holds only where it actually runs. Writing representative invalid cases is what turns the contract from something declared into something tested.

   Read chapter
3. 3

   Truncation under constrained decoding

   Constrained decoding removes malformed grammar, not every parse error. When output runs into `max_tokens` the response is valid JSON right up to the point where it stops and never closes, which is how 2% of responses start failing overnight with no change to the schema, the prompt or the model. The first place to look is the token ceiling and how long an output your schema is actually asking for.

   Read chapter
4. 4

   Required fields and invented values

   A required, non-nullable string is a standing claim that this field always has a value, so the decoder cannot emit null and the model supplies a plausible-looking name for an unassigned ticket. The same narrowing costs accuracy on ambiguous inputs: the hedge and the qualifying sentence are no longer legal continuations, so an enum forces a confident label earlier than the evidence supports. Losses concentrated in two categories out of forty point at the vocabulary, not the constraint.

   Read chapter
5. 5

   Evaluating a constraint change

   Before the constraint, the 3% that failed validation never reached the scorer; afterwards they are in the scored population, carrying whatever difficulty made them fail in the first place. A macro-F1 that moves across that change is partly the population moving rather than the model getting worse. Score both arms over the same fixed set of inputs, count a validation failure as a wrong answer instead of a missing row, and only then read the difference.

   Read chapter
6. 6

   Verifying schema enforcement

   A provider can accept the same `response_format` field, ignore it, and answer 200; the only symptom is a validator failure rate that reads like a model regression. Establish enforcement per provider-and-model pair with a canary the non-enforcing path must fail, fall back to a prompted mode when a pair is unverified, and stamp every response with a marker recording which mode produced it.

   Read chapter
7. 7

   Latency and cost of constrained decoding

   Turning constraints on moves three numbers at once: output tokens fall because the preamble and fences are gone, p50 holds, and p99 rises as the grammar machinery adds per-token work. Finance reads the token saving, SRE reads the tail regression, and the two are not denominated in the same thing until you find the tightest deadline downstream and compute the exceedance rate on both arms.

   Read chapter