Output field order

A structured judge returns an object with at least two fields: the verdict, which is what the harness reads, and a rationale, which is what an engineer reads when triaging. Which of those two comes first in the schema looks like a formatting choice, and it reviews like one. It is not, because a decoder emits fields in order and every token it emits is a forward pass whose output sits in context for the tokens that follow.

That makes field order a question about computation rather than presentation. With the verdict first, the decision token is sampled directly from a single pass over the prompt and the rationale is generated afterwards with the verdict fixed. With the reasoning first, one to two hundred tokens of explicit work happen first, and the distribution over pass and fail is conditioned on all of it. What follows measures the difference, shapes the reasoning field so it does something useful, and covers the serialisation trap that silently reverses the arrangement.

Two lines in a schema#

class Verdict(BaseModel):            ## the version on main
    verdict: Literal["pass", "fail"]
    reasoning: str


class Verdict(BaseModel):            ## the version in the pull request
    reasoning: str
    verdict: Literal["pass", "fail"]

The objection that meets a change like this is reasonable on its face:

The model receives the entire schema in the request before it generates anything, so it knows both fields are coming. Reordering them is cosmetic. Happy to merge but I don't see what it buys.

The premise is true and the conclusion does not follow from it. The model does see both field names before generating, and what changes with the order is not what the model knows — it is how many forward passes happen between reading the answer and committing to pass or fail. In the first schema that number is zero. The decoder emits {"verdict": ", and the very next token is the decision, produced from a single pass over the prompt with no intermediate tokens to compute against. In the second, roughly 120 to 200 tokens of reasoning are generated first, each one a forward pass whose output is in context when the verdict token is finally sampled.

The reason that count matters is a property of the machine rather than of the prompt, and it is the load-bearing step in this chapter. A model does a fixed amount of arithmetic per token it produces — the same layers, the same width, the same number of operations, whatever the question is. There is no setting that lets it think harder about a difficult example than an easy one within a single token. So the only way to spend more computation on a decision is to produce more tokens before making it, and each of those tokens lands in the context where the next pass can read it. Tokens are the model's working memory as well as its output. A schema that puts the verdict first is a schema that allots one token's worth of arithmetic to the judgement and then a hundred and eighty tokens' worth to explaining it.

Naming what that buys plainly: reasoning first does not make the model smarter, it gives the same model more passes over the problem and somewhere to put the intermediate results. That is why the effect shows up as a change in accuracy rather than as a change in style.

This is the same left-to-right decoding that makes six fields in one call agree with each other, pointed at a different problem. There the issue is that earlier fields contaminate later ones. Here the issue is that the field you care about most is the one you have arranged to produce with the least computation behind it.

The same answer, judged twice#

Take one borderline output: an answer about a returns window that mentions "the returns policy" and paraphrases section 4.2 of the provided context accurately without quoting it. The criterion under test, C3, requires that the answer quotes or names the specific passage. Both schemas, same judge model, temperature 0.

{
  "verdict": "pass",
  "reasoning": "The answer refers to the returns policy, which identifies
    the source material it relied on, and the substance matches section 4.2
    of the provided context. This is sufficient identification of the source
    for the reader to locate it."
}

{
  "reasoning": "Criterion C3 requires the answer to quote or name the
    specific passage. Scanning the answer: it says 'according to the returns
    policy' and then paraphrases a 30-day window. The provided context
    contains a section headed '4.2 Returns and refunds'. The answer does not
    quote any span from it and does not name section 4.2 or the heading.
    'The returns policy' names the document, not the passage.",
  "verdict": "fail"
}

The first rationale is a defence of a decision that had already been made. It reaches for a standard the criterion does not contain — sufficient identification for the reader to locate it — because the verdict token was pass and the remaining text had to be consistent with it. The second rationale performs the check the criterion describes, in order, and arrives somewhere else. Neither model is more capable than the other; they are the same model given different amounts of room to work.

"Had to be consistent with it" is the same conditioning argument as before, pointed the other way, and it is worth walking rather than asserting. Every token of the rationale in the first transcript is drawn from a distribution computed over a context that already contains the word pass. Text in which a stated conclusion is followed by an argument against it is rare — arguments in documents run the other way, evidence and then verdict — so a continuation that undermines the verdict already emitted carries very little probability. The model is therefore extremely likely to produce supporting material, and if the criterion does not supply any, it will reach outside the criterion for some. The invented standard in the first transcript is not a lapse in reading comprehension. It is what the most probable continuation of "verdict": "pass" looks like when the criterion as written offers nothing that would justify a pass.

The second transcript is subject to the same pressure and it works in the team's favour, because the thing the verdict token is conditioned on is now an explicit walk through the criterion. By the time the decoder reaches "verdict":, the context contains the quoted phrase, the heading in the source, and the observation that neither the span nor the section number appears in the answer. A pass token after that text is the improbable continuation.

What the ordering does across a suite#

Running both schemas over 600 stored outputs, on single-criterion calls, separates the effect from anecdote:

Two things in that table are worth separating. The pass rate moved by 23 points, from 87% down to 64%, and by itself that proves nothing — a change that makes a judge harsher is not automatically a change that makes it right. A judge that failed everything would move the pass rate by more and be worthless. The kappa against human labels is what settles it, and the reason is that kappa is a statement about a relation rather than about the harness. A pass rate can be computed with no reference to anything outside the harness, so no value of it can be wrong; agreement with hand labels can only be computed against a standard, so it can be. The reasoning-first harness agrees with careful human reading far more often, which is what makes the 23 points failures the verdict-first harness was missing rather than false alarms it invented.

The flip rate is the other useful number, and it is the one to reach for when somebody asks how large a difference the suite can detect. A judge whose verdict changes on 9.2% of examples across repeated sampling of the identical input has that much irreducible noise in every comparison you run through it: 9.2% of a 400-example suite is about 37 examples that move between two runs in which nothing was changed. Reasoning first takes that to 3.8%, which on the same suite is about 15. Any release comparison run on this harness has to clear the movement those examples produce before it counts as a finding, so a regression that breaks twenty answers is invisible under the first schema and visible under the second, on the same suite, with no extra examples bought. The reasoning does not merely justify the verdict; it constrains it, because by the time the verdict token is sampled the context contains an explicit walk through the criterion and the distribution over pass and fail is much sharper.

The anchoring shows up in a way you can grep for, too. Across the 600 verdict-first transcripts, the rationale contradicts its own verdict in 0 cases — there is not one transcript where the model says pass and then explains why the answer fails. Human re-reading of a 60-example sample judged 11 of those verdicts wrong. A rationale that is never in tension with its verdict, in a system whose verdicts are wrong 11% of the time, is not evidence about the verdict; it is a text generated to agree with it.

That zero is the number to sit with, because a count of zero is what a genuinely independent second opinion never produces. If the rationale were a separate assessment that happened to be written down after the verdict, the two would disagree at roughly the rate any two assessments of a borderline case disagree, and some transcripts would say pass and then explain a failure. Getting none in 600 is only possible if the rationale is not a second assessment at all. The practical consequence for a triaging engineer is that reading the rationale cannot tell them whether to trust the verdict, since the rationale was written under the constraint of agreeing with it — which is a stronger statement than saying the rationales are unhelpful. They are informative about what the judge would say in defence of a verdict, and contaminated as evidence about whether the verdict is right.

Once the reasoning field is doing real work, it is worth shaping it. A single free-text field lets the model write whatever it likes, including a paragraph of summary that never touches the criterion. Splitting it into an evidence field and an analysis field pins it to the answer:

class Verdict(BaseModel):
    quoted_evidence: str   # verbatim span from the answer, or "" if none
    analysis: str          # how the span does or does not satisfy C3
    verdict: Literal["pass", "fail"]

Requiring a verbatim span first forces a retrieval step through the answer text before any evaluation happens, and it gives the triaging engineer a pointer instead of a paraphrase. On the same 600 outputs this moved kappa against the human labels from 0.83 to 0.88 and, more usefully, made 41 disagreements resolvable in seconds because the evidence field showed the judge had quoted the wrong sentence.

The word "verbatim" is where the gain comes from, and it is easy to write the field without it and get nothing. A paraphrase can be generated from the model's impression of the answer, so a field asking for "the relevant part of the answer" is satisfied by more of the same prose that was going to be produced anyway. A verbatim span cannot be: producing it requires locating the characters in the answer and copying them out, so the tokens that land in the field are constrained by the text rather than by the impression. That is also what makes the field auditable afterwards, since a span that does not appear in the answer is a mismatch anybody can detect with a substring test, and a paraphrase that drifts from the answer is a judgement call. The same property is doing both jobs: tying the judge's first act to the literal text is what improves the verdict and what makes the disagreement resolvable.

Making sure the order you wrote is the order that is emitted#

The mechanism depends entirely on the model actually emitting the fields in the order you declared, which is a property of the serialisation path rather than something to assume. Structured output implementations that constrain decoding against a JSON schema generally emit object keys in the order the properties are declared, and that is the behaviour you are relying on. Implementations that ask for JSON in the prompt and parse whatever comes back do not guarantee it, and a schema library that serialises properties alphabetically will silently order them by name rather than by the order you declared them in.

Alphabetical ordering is worth dwelling on because of how quietly it goes wrong. With these two field names it happens to give the arrangement you wanted, since reasoning sorts ahead of verdict, and a harness can run for months on an accident of spelling. Rename the rationale field to analysis and the accident still holds; rename it to summary or why and it does not, and the judge starts committing before it works. Nothing about that failure is loud. The generated object still has both keys, the parser still returns both values, the types still check, and the only visible symptom is a metric that moved.

The check is to log a raw response and look at the string before it is parsed, because the parsed dictionary tells you nothing — key order is gone by then in most languages, and preserved-but-meaningless in the ones that keep it.

raw = client.responses.create(model=JUDGE_MODEL, input=prompt,
                              text_format=Verdict).output_text
assert raw.index('"reasoning"') < raw.index('"verdict"'), raw[:200]

That assertion belongs in the harness's own test suite, not in a comment, because the failure mode is silent: everything parses, the dashboard fills in, and the judge is anchoring again.

Where reasoning-first stops being the whole answer#

Reasoning models change the arithmetic, and the reason is that they were already buying the thing this chapter is about. When a model performs extended internal reasoning before producing its response, those internal steps are tokens too, and they are generated before the first visible output token, so the count of forward passes standing between the answer and the verdict is large under both schemas. Field order was never the mechanism; the number of passes was, and the model now supplies them regardless. The ordering still matters for the visible fields, because a visible rationale generated after a visible verdict is still conditioned on it, but the gap between the two schemas narrows considerably — in the same suite, run on a reasoning model, C3 came out at 71% verdict-first and 68% reasoning-first, a three-point spread where the ordinary model produced twenty-three. The cost profile moves the other way, since thinking tokens are billed and the internal trace is not always available for you to read, which matters for a judge whose rationales are the artefact your engineers actually use when triaging failures.

The honest limitation is that reasoning-first buys computation, not truthfulness. A rationale that precedes the verdict is still a generated text and is not a faithful transcript of whatever the network computed; work on chain-of-thought faithfulness has repeatedly shown models producing plausible reasoning that does not reflect the features actually driving the output, including cases where a hint injected into the prompt changes the answer and never appears in the explanation. What the ordering guarantees is that the verdict is conditioned on a substantial intermediate computation, and that the text you read is at least the text the verdict was conditioned on rather than a defence written afterwards.

None of this rescues a criterion that was never well specified. A reasoning-first judge asked whether an answer is "helpful" will produce 180 tokens of thoughtful prose and then a verdict that flips as often as before, because the reasoning has no threshold to reason toward. The ordering multiplies the quality of the criterion rather than substituting for it.

Field order, then, is the cheapest change in a judge harness and one of the largest: two lines, no new calls, and a verdict that goes from being sampled cold to being sampled after an explicit walk through the criterion. It is also genuinely purchased rather than free. Verdict-first schemas exist for reasons — documentation examples put the important field first, max_tokens can be capped low, a streamed response can be aborted once the field you need arrives, and the p50 latency is four times better. Every one of those benefits is real, and every one of them is paid for by removing the computation that makes the verdict worth having.

The harness now has one criterion per call and reasoning ahead of the verdict, and the numbers coming out of it are stable enough to compare across releases. What has not been examined is which model is doing the judging, which turns out to matter more than any of the structural choices when the judge and the thing it grades come from the same place.