One criterion per call

The obvious implementation of a five- or six-criterion rubric is one call per answer with a JSON schema carrying one boolean field per criterion. It is a sixth of the cost of the alternative, it is one round trip, and it is what every harness starts as. What it produces is not six measurements of an answer; it is one reading of the answer, serialised into six fields that were generated in order, each with the previous ones in front of it.

That distinction has a consequence you can measure. Criteria describing unrelated properties come back correlated, strongly enough to send a team to work on the wrong problem. This chapter separates the association the call structure creates from the association the answers genuinely have, and prices the split that removes it.

The cross-tabulation that starts the investigation#

Two criteria from a six-criterion support-answer rubric, cross-tabulated over 600 graded outputs from a single-call harness. C1 asks whether the answer addresses the user's actual question; C3 asks whether it quotes or names the passage it relied on.

                       C1 answers_question
                        pass       fail
C3   pass                358          8      (366)
cites  fail              161         73      (234)

Among the 366 answers that cite a source, 2.2% fail C1. Among the 234 that do not, 31.2% fail it. Both figures come out of the table by dividing one cell by its own row total: 8 failures among the 366 citing answers is a bit over two in a hundred, and 73 failures among the 234 non-citing ones is a bit under a third. The division by the row total is what makes the two comparable, since the rows hold different numbers of answers and the raw failure counts — 8 against 73 — would confound the failure rate with how many answers landed in each row.

Dividing that way is what "conditioned on" means in this context. It asks: given that I already know an answer's C3 verdict, what does that tell me about its C1 verdict? If the two criteria were describing unrelated properties the answer would be nothing, and the two rates would come out roughly equal. Instead one is about fourteen times the other. Citation and question-answering are not obviously related properties — an answer can address exactly the question asked and simply not point at a passage, and in this material that combination is common because the assistant paraphrases. A fourteen-fold difference in C1 failure rate conditioned on C3 is a strong claim about the answers, and reading forty of the 161 conflicted cases by hand found thirty-three that plainly answered the question. The judge had marked them down anyway.

Why one call produces agreeing fields#

The six verdicts come out of a single autoregressive generation. The model reads the answer, the context and all six criteria, and then emits tokens in order: the value of the first field, then the second, and so on to the sixth. Every field after the first is generated with all the preceding fields already in the context window, so field five is conditioned on the four verdicts that precede it in exactly the way any token is conditioned on the tokens before it. Nothing in the architecture offers a way to compute the fields in parallel or in isolation.

The word "conditioned" is doing real work there, and it is worth unpacking once, because the whole chapter rests on it. A model produces text one token at a time, and each token is drawn from a distribution computed over every token that came before it in the sequence. That is not a metaphor for influence; it is the definition of the computation. When the decoder reaches the position where field five's value goes, the string it is extending literally contains "c1": false, "c2": false, "c3": false, "c4": false, and the distribution over true and false at that position is a function of that string. In ordinary text a run of four identical values makes a fifth identical value more likely — that is a statistical regularity of the documents the model learned from, where lists of verdicts about one object tend to be consistent — and the JSON object being emitted here is such a document. The model is not deciding to agree with itself. It is continuing a text that has begun to look a particular way.

There is a second mechanism stacked on the first. Before any field is emitted, the model has processed the answer once and represented it — the internal state that produces the first verdict already encodes something like an overall assessment, because that is what a model reading a piece of text and being asked to grade it computes. The six fields are then decoded from that shared state, so even the first two fields, with no verdict tokens between them, are drawn from the same impression. Splitting the call does not eliminate this second mechanism; it stops each verdict from being conditioned on the previous ones in text.

Constrained decoding does not help here, and it is worth being clear about why, since teams often assume a JSON schema imposes some kind of field independence. A schema constrains which tokens are legal at each position. It does not change the distribution over the legal ones, and the distribution over true and false for field five is computed from a context that contains fields one through four.

The step people skip is what constrained decoding actually does to the numbers. The model produces a score for every token in its inventory; the constraint sets the score of every token the grammar forbids to nothing, and then the remaining scores are divided by their own new total so that they add up to one again. That division scales every surviving score by the same factor, so the ratio between true and false after the constraint is exactly the ratio it was before. If the context has already pushed the model to four parts false for every one part true, forbidding maybe and null and every other string leaves it at four parts to one. Constrained decoding guarantees that the output parses. It has no mechanism for changing what the output says, which is why a schema is the wrong place to look for independence between fields.

Measuring whether the correlation is real#

The test is to run both harnesses over the same 600 stored outputs and compare the association between criterion pairs. The phi coefficient is the right statistic for two binary variables, and it is Pearson correlation applied to 0/1 columns — a value near 0 means the two verdicts move independently of each other, and a value near 1 means they rise and fall together — so a numpy one-liner does it.

Two design decisions inside that sentence are what make the result mean anything. The outputs are stored, so both harnesses grade byte-identical text and no difference between them can be a difference in the answers. And the comparison is between two columns of the same criterion pair rather than between two pass rates, because a pass rate can be identical under both harnesses while the answers that fail are completely different — which is what happens here, and what the aggregate later in this chapter conceals.

Writing each verdict as 1 for pass and 0 for fail is what lets an ordinary correlation be computed over them at all, and it is worth seeing why that is legitimate rather than a trick. Correlation asks whether two columns rise and fall together relative to their own averages. With a 0/1 column the average is just the pass rate, an answer that passes sits above it and one that fails sits below it, so the products that go into the correlation are positive when the two criteria agree about an answer and negative when they disagree. Summing them gives the same thing it always gives — the degree to which knowing one column tells you about the other — with the pass rates divided out, which matters here because C6 passes almost everything and C1 does not. Because it is an ordinary correlation, an ordinary correlation routine computes it, and a numpy one-liner is the whole implementation:

import numpy as np

def phi(a: np.ndarray, b: np.ndarray) -> float:
    """a, b: 1-D arrays of 0/1 verdicts over the same examples."""
    return float(np.corrcoef(a, b)[0, 1])

single = load_verdicts("run_single_call.jsonl")   # shape (600, 6)
split   = load_verdicts("run_six_calls.jsonl")    # shape (600, 6)
human   = load_verdicts("human_labels.jsonl")     # shape (60, 6)

for i, j in [(0, 2), (1, 3), (0, 4), (2, 5)]:
    print(NAMES[i], NAMES[j],
          round(phi(single[:, i], single[:, j]), 2),
          round(phi(split[:, i], split[:, j]), 2))

Running that over the six-criterion rubric gives a table that separates two very different things:

Three of those four associations collapse when the criteria are asked separately, and the split values land close to what the human labels say. One of them does not move: grounding and fabrication stay correlated at about 0.55 under every instrument, including the humans, because an answer that invents an entity is by construction making a claim the context does not support. That correlation is a fact about the answers and you want to keep it. The value of the split is that it tells the two kinds apart, which no amount of staring at the single-call numbers can do — 0.62 and 0.57 look like the same thing until you re-measure.

The human column is what turns that comparison into evidence, and leaving it out is the way this experiment is usually run badly. On its own, a phi that falls from 0.62 to 0.18 shows only that two harnesses disagree; the split could as easily have destroyed a real association as removed an invented one. The human labels break the tie because they were produced by an instrument that shares no mechanism with either harness — no shared decoding pass, no shared impression of the answer, no ordering of fields — so an association that survives in them is one the answers actually have. Read the table with that in mind and the first row says the split moved toward the truth, while the second says grounding and fabrication were correlated all along and every instrument found it. Sixty labelled examples is a small sample and the column carries real uncertainty; what it is being asked to do is separate 0.15 from 0.62, which is a gap wide enough that sixty is enough.

The practical consequence in this suite is 128 examples. That is how many answers were marked as failing C1 under the single call and passing it under the split, which is close to a fifth of the entire 600 carrying a failure attributed to the wrong requirement. Set that against the C1 failure counts and the accounting closes: 132 answers failed C1 under the single call and 11 under the split, so of the original 132 all but four were rescued by asking the question on its own, and the split found seven failures of its own that the single call had missed.

When the headline number does not move#

A reasonable objection is that the aggregate barely changed. The all-criteria pass rate was 48% under the single call and 51% under the split, a three-point difference on 600 examples, and if that is the number on the gate then the whole exercise bought three points of a figure nobody trusts to a point anyway. The objection is correct about the aggregate and wrong about what the suite is for.

The composition underneath the aggregate changed completely. Under the single call, 234 failures were attributed to C3 and 132 to C1; under the split, 231 were attributed to C3 and 11 to C1, with the difference redistributed across C5 and C2. Roughly the same answers fail, and the reason recorded against each of them is different for a fifth of the suite. Every decision downstream reads the reason: which criterion to invest in this quarter, which regression to block a release on, which twenty failures an engineer pulls up when they sit down to improve the retriever. A gate that says yes or no is the smallest thing an evaluation harness does, and it is the only thing an uncorrelated and a correlated harness agree on.

The belief that makes this hard to see is the one worth naming, because it is reasonable and everyone holds it before they run the experiment. Six boolean fields in a response object look like six measurements. They are typed separately, they are named separately, they are stored in six columns, they are charted as six lines, and every downstream consumer treats them as six facts about the answer. The step that fails is the one before all of that: six measurements requires six acts of measuring, and there was one. Producing six numbers from one reading is a formatting operation, and formatting does not create information. Where the six columns look independent, the independence is something the schema asserted rather than something the harness performed, which is why surfacing the association between pairs of columns is the check that settles it rather than reading any single column more carefully.

There is a case where the aggregate does move, and it is the one that costs money. When correlation inflates failures on a criterion that is already failing, the all-pass rate is roughly unchanged because those answers were failing something anyway. When it inflates failures on a criterion that is otherwise clean, answers that pass everything else get pulled below the line, and the gate blocks releases for reasons that are artefacts of the call structure.

The instruction that looks like it fixes this#

The response that occurs to nearly everyone, because it costs nothing and reviews well, is to keep the single call and add a line to the prompt:

Evaluate each criterion independently. Do not let your verdict on one
criterion influence your verdict on any other. Judge each on its own merits.

Measured on the same 600 outputs, that instruction moved phi(C1, C3) from 0.62 to 0.57. The reason it barely moves is structural: the instruction asks the model to ignore information that is physically in its context at the moment the later fields are decoded. There is no mechanism by which a sentence in the prompt removes the preceding verdict tokens from the attention window, and the shared-impression effect is untouched by it, since the impression was formed before any field was emitted. The instruction is not lying — the model is genuinely attempting to comply — but compliance would require a computation the forward pass cannot perform.

Spelling out what full compliance would take is the clearest way to see that it is not on offer. To judge C5 without reference to C1 through C4, the model would have to compute the distribution over true and false at that position as though four verdict tokens were not in the sequence. The sequence it is actually extending contains them, and a forward pass computes a distribution over the sequence it is given; there is no argument to it that says which of the preceding tokens to disregard. Producing a genuinely unconditioned verdict for C5 means running a pass over a sequence that does not contain the other four, which is another way of describing the split. The instruction and the split are asking for the same thing. One of them changes the input the model is given, and the other adds a sentence to it and hopes.

The instruction is worse than doing nothing, in one specific respect. After it is merged, the prompt contains a line asserting that the criteria are judged independently, so the next person to investigate a suspicious correlation reads that line, concludes the concern is already handled, and looks elsewhere. The harness is exactly as contaminated as before and now carries documentation claiming otherwise.

Separate calls leave the model and the answer shared#

Splitting removes the textual conditioning between verdicts and reduces the shared-impression effect, and it leaves a good deal in place. Every call still uses the same judge model with the same systematic tendencies, so a model that is generally lenient about citation will be lenient about it in isolation. Every call still sees the same answer, and a persuasive, confidently written answer will pull all six verdicts in the same direction through the same mechanism operating six times independently. The split changes the correlation induced by the call structure; it does not make the six measurements independent in any strong sense.

Splitting also does nothing for a criterion that was badly written. C5, the refusal criterion, had a kappa of 0.44 between judges before the split and 0.46 after, because its problem is that "the context does not contain the answer" was never defined for partial coverage. Six calls of an ambiguous question produce six ambiguous answers.

The cost is the part teams underestimate. Six calls per answer over 400 golden examples — the fixed, curated inputs the suite replays on every run, stored so that two runs are comparing the same work — is 2,400 judge calls per suite run instead of 400, and the latency goes from one round trip per example to six unless you fan them out, which most harnesses do not do by default. On a suite that runs on every pull request, that multiplication is the difference between a two-minute check and a twenty-minute one.

The token bill grows faster than the call count, and knowing why is what makes the trade negotiable. Splitting does not divide the prompt six ways. Each of the six calls has to carry the answer and its retrieved context in full, because a judge cannot check a criterion against material it was not sent, so the bulk of the input is sent six times rather than once and only the criterion text differs between them. Six calls therefore cost close to six times the input tokens as well as six times the calls, and the output tokens multiply too, since each call writes its own rationale. That is the shape that makes the two-call grouping above attractive: three criteria sharing one call share one copy of the context, so grouping trades a known amount of contamination for a known amount of money rather than guessing at either.

The measurement to keep, then, is the pairwise association under both harnesses. It costs one extra run over stored outputs, and it is the only thing that distinguishes a correlation you should preserve from one your implementation invented — a difference that shows up not in the pass rate but in which requirement a fifth of your failures are filed against. Which criteria deserve their own call and which can share is a budgeting conversation, taken up later in this topic.

Having decided that each criterion gets its own call, there is still a decision inside each of those calls: what the model emits, and in what order. A single-criterion call that returns {"verdict": ..., "reasoning": ...} has the verdict produced before any of the reasoning exists, which is the subject of the next chapter.