Required fields and invented values

A schema is usually written on a good day, from a happy-path example, and the instinct is that a record where every field is present is easier to consume than one full of optional values. Under constrained decoding that instinct has a cost the validator can never report, because the values it produces are all the right type.

The mechanism is the same one that removes fences and out-of-enum labels: the grammar masks illegal tokens before sampling. What changes is that null is an illegal token, and the model's belief that a field is empty is expressed by exactly that token. This chapter is about what happens to that belief when it has nowhere to go, how to encode absence so it survives the round trip, and why the same narrowing also costs accuracy on inputs where the evidence is thin.

A constraint the model satisfies by inventing#

Consider the decoding step immediately after {"ticket_id":"91443","urgency":"low","assignee": has been emitted. Left alone, the model's distribution over the next token on an unowned ticket concentrates on null, because the training data is full of documents where an absent value is written that way and the context contains no name. If assignee is typed string and marked required, the grammar admits a quote and nothing else. The mask sets null to zero probability, and the sampler renormalises over what remains, meaning the surviving candidates are rescaled so their probabilities sum to one again:

position: after '"assignee":'

unconstrained            after the schema mask
  null      0.62           "         1.00   (only legal token)
  "         0.31
  \n        0.04           renormalised over string starts:
  0         0.01             "M        0.19
  ...                        "Support  0.14
                             "j        0.11
                             "Un       0.09
                             ...

The model's belief that the field is empty has nowhere to go. It cannot be expressed, so the 0.62 of probability mass sitting on null is redistributed across the string continuations that were previously the minority view, and the most likely of those is a name. The schema did not ask the model to guess; it removed the vocabulary in which "I don't know" is sayable, and a fabricated value is what is left.

Renormalising is worth doing by hand once, because the word hides the operation that does all the work here. A sampler needs a probability distribution, and a distribution has to sum to one. Zeroing null, the newline and the digit leaves a single survivor at that position — the opening quote, holding 0.31 of the original mass — and 0.31 is not a distribution, because it sums to less than one. The repair is a division: every surviving candidate is divided by the total the survivors hold between them. With one survivor holding all of the remaining mass, that division sends its probability to 1.00. The same division runs again one step later, among the string continuations, which is where the weights on "M and "Support come from.

Two consequences follow, and the second is the one that gets missed. The first is that the 0.62 does not go anywhere: mass zeroed in one place reappears proportionally across the others, because that is what dividing by a smaller total does. The second is that the mask cannot reorder the tokens it left alone. Whatever the model preferred among string starts before the schema was applied, it prefers in the same order after, at larger numbers. So the model has not become careless and has not started guessing in any sense it could report on. It has been asked to choose among options it had already ranked, with the option it actually wanted deleted from the list.

The closest familiar thing is a NOT NULL column with no default. In Postgres, an insert that omits the value fails, loudly, at the boundary, and the application deals with it. A required non-nullable field in a constrained-decoding schema enforces the same rule with the opposite failure mode: instead of rejecting the row, it makes one up, and hands it back with the same shape and the same confidence as a row that was real. That inversion is what makes the bug hard to find, because every instrument you have is measuring shape.

That comparison is a reframe rather than a full analogy, and the place it stops holding is the place the teaching is. A database rejects because it has somebody to reject: the insert arrived from a caller, the caller is still waiting, and handing an error back to it is a legal move. A decoder halfway through a response has no caller to answer and no error to return. The only action available at that position in the grammar is to emit a token. So a constraint the model cannot satisfy honestly does not turn into a failure the way the database version does; it turns into an emission, and the emission is well-typed because the grammar made sure of that first.

Encoding absence so it survives the round trip#

Three encodings are in common use and they are not interchangeable.

The union with null is the default, because it puts the token the model wanted back into the legal set. The third row is worth the extra field on anything that feeds a decision. There is a real difference between this ticket has no owner and I could not tell from this email who owns it, and a single null flattens them into the same value. If a human has to review the unclear ones and does not have to review the genuinely unassigned ones, that distinction is the difference between a 9% review queue and a 100% one.

Catching invention when every response validates#

None of the instrumentation that finds truncations would find this, because nothing here raises. What does find it is a check that compares a generated field against the set of values it is supposed to refer to, which for an assignee is a table the company already has:

select date_trunc('day', created_at) as day,
       count(*) filter (
         where assignee is not null
           and lower(assignee) not in (select lower(display_name) from staff)
       ) as unknown_assignee,
       count(*) as total
from extractions
group by 1
order by 1 desc;

That query runs in about 40 ms over a day of rows on this service and fires on the first day, because the legal set is closed and known and the only defensible count is zero. Almost every extraction schema has at least one field of this kind — an assignee, a product code, a customer id, an account name — and none of them are checkable by the schema itself. Inlining the legal set as an enum so the illegal value is unreachable works when the set is small and stable; a directory of 1,900 names that changes weekly is neither, so the check has to run after the fact.

For fields with no such table, the direct measurement is a probe set. Take 200 tickets, strip the line that carries the fact, keep the gold answer as absent, and measure how often the model produces a value anyway. On an eleven-field schema with every field required and non-nullable, that probe returned a name on 190 of 200; after the union with null it returned one on 19. Those two counts share a denominator, so they compare directly: nineteen probes in twenty produced a name for a fact that had been deleted from the input, and after the change roughly one in ten did. Running it in CI against a recorded set of inputs costs a couple of dollars a month and turns a property that is otherwise invisible into a number that can regress a build.

The reason the probe has to delete the line rather than sample production is that otherwise there is no gold answer to compare against. On a real ticket you do not know whether the owner was stated somewhere in a forwarded thread, so a name in the assignee field is unfalsifiable — it might be an extraction and it might be an invention, and nothing in the record says which. Stripping the fact yourself is what makes absent a known correct answer, and it is the only reason the resulting number means anything. That is the general shape of a probe for invented values: construct the case where the honest answer is nothing, and count how often you get something.

The invented value with no directory behind it#

The second version of this failure is worse to find, because the fabricated value looks like every other value in the column. The same schema marked due_date required and non-nullable, and roughly 1,000 tickets a week arrived with no date stated anywhere in the body. Every one of them came back with a date, all of them well-formed, all of them in the future, none of them extracted from anything.

The tell is the tidiness. A field carrying values genuinely lifted out of the input inherits the messiness of the input, and on the 400 tickets that did state a date the same field matched the stated date on 92% of them with the seven-day cluster accounting for 3%. Two distributions come out of one column, one shaped by the source documents and one shaped by the model's priors about when work is usually due, and the distance between them is the fabrication rate.

The comparison worth making is between the same cluster on the two halves of the traffic, because that is the only pair of numbers with everything else held fixed. On tickets stating no date, the value exactly seven days out took 61% of responses. On tickets that did state one, the same value took 3%. That is about twenty times as common when there is nothing in the input to copy, on the same field, the same schema and the same model — so the difference cannot be a property of any of the three. The only thing that differs between the halves is whether a fact was there. The ratio is the measurement, and the 61% on its own is not one: a business where most work genuinely falls due in a week would produce a large seven-day share honestly, and would produce it on both halves. This generalises to any generated scalar for which you have no reference table: split the traffic by whether the fact is present in the input, plot the value distribution for each half, and a field that is being invented will show a spike where a real field shows a spread.

The same narrowing, applied to an enum#

The mechanism that fabricates a name also loses accuracy, and the clearest demonstration came from a classification service picking one of 40 categories. Constrained decoding went on with the categories as an enum. Validation failures went from 3% to zero, and macro-F1 over the held-out set — the labelled tickets kept out of every tuning decision — dropped from 0.712 to 0.681, with p99 latency up 80 ms. Macro-F1 is the accuracy measure this argument runs on: for each category you compute precision, the share of the predictions for that category that were right, and recall, the share of that category's tickets it actually found; their harmonic mean is that category's F1, and the forty F1 scores are averaged with every category weighted equally, so a label with 40 tickets counts as much as one with 400. Almost the entire loss sat in two categories: billing_dispute and billing_adjustment.

Two things are happening at once, and separating them is the whole diagnosis.

The first is that the constraint removes continuations the model would otherwise have used. Before the enum, on an ambiguous ticket, the model would often write something like "This looks like a billing issue; the customer is contesting a charge rather than requesting a correction, so" and then commit to a label. Those tokens are not decoration; they are computation, and the label that follows them is conditioned on them. Under a bare enum the first legal token is the first token of a category name, so the model commits immediately, on the ticket where it least should. The accuracy cost is concentrated exactly where you would expect — the hardest inputs, where the evidence is thin — and it is invisible in the validation metrics, which are all now perfect.

The fix for that half is to make the hedge legal again by putting it in the schema, before the label, since a constrained decoder produces properties in schema order:

class Classification(BaseModel):
    evidence: str = Field(description="the phrases in the ticket that determine the category")
    reasoning: str = Field(description="one or two sentences weighing the plausible categories")
    category: Literal["billing_dispute", "billing_adjustment", ...]   # 40 members
    confidence: Literal["high", "medium", "low"]

Ordering category last is the point. The model writes free text under a string grammar, which imposes almost no restriction, and only then meets the enum, by which time the label is conditioned on the reasoning it just produced. On this service, that ordering recovered 2.4 of the 3.1 lost F1 points, at a cost of roughly 60 extra output tokens per call. Declaring the same four fields with category first recovers nothing, because a field the model has not written yet cannot condition anything.

Three steps sit under that and each is checkable on its own. Decoding is append-only: every token the model emits joins the context, and every token chosen after it is conditioned on everything already sitting there. A string-typed field imposes almost no grammar, so at that position the model may write out the weighing of the two billing categories it would have written unprompted. When the decoder then arrives at category, the enum mask is applied to a distribution that was computed with those sentences in its immediate left context. "Conditioned on the reasoning" is therefore not a figure of speech: the reasoning tokens are literally inputs to the forward pass that produces the label.

That is also the answer to the obvious cheaper proposal, which is to put "think carefully before choosing" in the system prompt and save the tokens. It does not work, and the reason is mechanical rather than a matter of degree. An instruction in the prompt cannot put tokens into the response, and the response is where this conditioning happens. Only a field in the schema creates a position at which the model is obliged to write something down before it commits to a label, which is why the fix has a token cost at all, and why the ordering rather than the wording is the part that recovers the points.

The second thing happening is the geometry of the loss. Thirty-eight categories were unaffected and two lost almost everything, which is not the signature of a constraint that is too tight in general. If grammar-constrained decoding were broadly harming the model on this task, the damage would be spread across categories in rough proportion to their difficulty. Damage concentrated in one adjacent pair says those two labels are the problem: they are near-synonyms whose boundary the enum forces the model to resolve without giving it any definition of where the boundary lies. The enum ships the strings and nothing else — no description, no examples, no tie-breaking rule — so the model resolves the ambiguity by prior, and the prior favours whichever of the pair is more common in training-like text.

That reading points at fixes in the vocabulary rather than at rolling back the constraint. Merge the two categories into one and split downstream on a separate boolean if the distinction matters operationally. Or rename them so the strings themselves carry the distinction — billing_dispute_customer_contests_charge against billing_adjustment_we_issue_credit — which costs a few tokens per response and gives the decoder labels whose surface form encodes the rule. Or add an other member and a confidence field, and route low-confidence billing predictions to a human. All three are cheaper than giving up a 3% validation-failure rate.

The constraint changes reachability, not capability#

Constrained decoding is not making the model less capable. The mask changes which tokens are reachable at each step; it does not change the weights, the context, or what the model believes. On unambiguous inputs the constrained and unconstrained outputs agree almost perfectly, because the model's preferred continuation was already inside the legal set and masking a set of tokens it was not going to pick has no effect. The gap opens only where the model's mass was outside the legal set, which is the definition of an input the schema and the model disagree about.

It is also not a hallucination fix, and it is occasionally sold as one. A schema guarantees that ticket_id is a string and that urgency is one of three values; it has no opinion about whether the ticket id is the one in the email. The 15 responses in the earlier fixture run that were schema-perfect and semantically wrong do not respond to constraining harder — as this chapter shows, over-constraining raises that number, because every field you mark required is a field the model must fill whether or not the input supports one.

The honest limitation is that the trade between abstention and coverage does not have a right answer that a schema can express. Nullable everything and you push the decision to a downstream consumer that now has to handle eleven optional fields; require everything and you get invention. What the schema can do is make the model's uncertainty representable — through nulls, sentinel members, an explicit status field, a confidence level — so that the decision is made by code that can see the evidence rather than by a mask that cannot.

That is the discipline this chapter asks for: read every required, non-nullable field as a promise that the input always contains the fact, and check that promise against the traffic rather than against the example the schema was written from. Where the promise does not hold, the model will keep it anyway, and it will do so in a form no validator can reject.

None of that tells you whether the 3.1 points of F1 were worth it, and the number itself is less solid than it looks. Both figures came from the same eval harness, run before and after, over the same set of tickets — and the constraint changed which tickets the harness was able to score at all.