Same-family judges

Picking the judge model is usually not a decision anybody makes. The assistant runs on model X, the judge becomes X's larger sibling, and the reasons never need defending: one contract, one SDK, one invoice line, one rate limit to manage. The harness may be well built by then — one criterion per call, reasoning before the verdict — and still be reporting a number that a room full of experienced people would not recognise.

The size of a judge-versus-human gap is evidence for nothing in particular, because at least three different mechanisms produce the same shape and they have completely different fixes. This chapter is about telling them apart before spending on any of them: what each explanation predicts, the frozen-set design that lets four instruments be compared at all, and what the decomposition looks like when it comes back.

Three explanations that fit the number#

The first is that the judge and the humans were never grading the same thing. Agents asked whether an answer was acceptable fold in tone, length, whether it opened with an unnecessary preamble, and whether they would have been comfortable sending it. The judge was asked five specific questions. Two instruments measuring different constructs will disagree, and the disagreement carries no information about either one's quality. Sampling belongs in this category too: if the human-labelled outputs were pulled from a triage queue rather than drawn at random from the suite, the human number is a number about hard cases.

The second is that the rubric text is more lenient than the people who wrote it believe. A criterion saying every factual claim must be supported by the provided context is read by an agent as meaning the context states it, and by a judge as accepting a claim the context makes plausible. The judge is applying the rubric faithfully, and the rubric does not say what its authors meant. This failure produces a gap that has nothing to do with which model is judging.

The third is family preference. A generator and its larger sibling were pretrained on overlapping corpora and shaped by the same post-training recipe, the same preference data conventions and the same house style guidelines, so they share opinions about what a good answer looks like: how long it should be, how much hedging is appropriate, whether to use a bulleted list, how to open, how confident to sound. When the judge grades the generator's output, the output already looks the way the judge would have written it, and formal similarity reads as quality to a judge whose notion of quality was tuned on the same data.

The objection raised against the third explanation, almost every time, is that the judge could not possibly be exhibiting self-preference because it never produced these outputs — it is a different model, a larger one, which has never seen its own text here. The effect does not require a model to recognise its own writing. It requires shared preferences over form, and siblings have those by construction.

Laying the steps out separately is what makes the objection dissolve, because the objection is answering a claim nobody made. Post-training shapes a model by rewarding some continuations over others, and the rewards encode preferences about form as well as about content: this length, this amount of hedging, this opening, a list here rather than a sentence. Two models tuned on the same preference data and the same house style guidelines therefore end up with the same preferences over form. One of them is later asked to write support answers, so it writes them in that style. The other is later asked to grade them, and it grades using the same preferences, which now sit on the grading side as an implicit view of what a careful answer looks like. Nothing in that chain requires either model to notice anything about the other, and no step in it is a step of recognition. Recognition is a separate mechanism that also exists, and the objection retires only that one.

The mechanism is often more direct than shared pretraining: small production models are frequently distilled from their larger sibling, which means the generator was explicitly trained to imitate the judge's output distribution. Distillation is worth stating in the concrete, since it is the strongest version of the argument. Training a small model by distillation means generating text with the large one and fitting the small one to reproduce it, so the small model's objective is literally to place high probability on the strings its larger sibling would have produced. When that generator writes a support answer and its teacher grades it, the answer is a sample from a distribution the grader was used to define. Under distillation, "the judge prefers text that looks like what it would produce" and "the generator produces text that looks like what the judge would produce" are the same sentence.

The research literature separates the two ideas. Work on self-preference in LLM evaluators has found that models score their own generations higher, and that the effect correlates with a model's ability to recognise its own outputs, which establishes self-recognition as one pathway. It does not establish it as the only one, and the stylistic-similarity pathway needs no recognition at all — a judge that likes three-sentence answers with a citation in the second sentence will reward any model that writes that way, sibling or not.

Separating them costs less than rewriting the rubric#

The three explanations make different predictions, and the measurements that discriminate them are cheap relative to what is at stake.

Order matters, and it is the reverse of how interesting the hypotheses are. The bookkeeping check is nearly free and can eliminate a slice of the gap before anyone touches a prompt.

The frozen set is the load-bearing part of the design. All four instruments have to see the identical 60 outputs with the identical contexts, and the outputs must not be regenerated between runs, or a change in the generator's sampling contaminates every comparison. Store the answers and contexts as files and have every run read from them.

FROZEN = load_jsonl("eval/frozen_60.jsonl")   # answer + context, never regenerated

for name, judge in [("same_family", judge_x_large),
                    ("cross_family", judge_other_vendor)]:
    rows = [judge(ex, criterion=c) for ex in FROZEN for c in CRITERIA]
    write_jsonl(f"eval/out/{name}.jsonl", rows)

Running it on one support-assistant suite produced a decomposition rather than a verdict:

Read from the bottom, 5 points of the gap were the two instruments grading different constructs, 9 points were the rubric text being more permissive than its authors intended, 7 points were family preference, and 2 points are residual disagreement of the kind any two graders have.

Those four slices are differences between adjacent rows, and the reason they add up to the whole gap is arithmetic rather than luck: 94 down to 87 is 7, 87 down to 85 is 2, 85 down to 76 is 9, and 76 down to 71 is 5, and the four differences between five numbers necessarily total the distance from the first to the last, which is the 23 points the blinded review produced. What makes each difference mean something is the design of the ladder rather than the subtraction. Every adjacent pair of rows differs in exactly one thing. Between the top two rows only the judge model changes, and the criteria, the outputs and the contexts are identical. Between the second and third only the grader changes from a model to a person, with the same rubric text applied literally by both. Between the third and fourth the same people change how they read the rubric, from its letter to their own per-criterion judgement. Between the fourth and fifth the same people change what they are being asked, from five criteria to overall acceptability. One variable per rung is what licenses attributing a rung's difference to that variable, and it is the reason the frozen set has to be built beforehand rather than assembled as each measurement is taken.

Nothing about that arrangement makes an individual difference precise, and the seven points are a good place to see why. A net difference of 7 points across 60 outputs is only about four answers. Eleven outputs actually moved from pass to fail when the judge changed family, which is more than four, and the rest of the movement is answers going the other way and cancelling in the total. A net shift and a count of disagreements are different quantities, the second is always at least as large as the first, and quoting the first while investigating the second is how a decomposition gets over-read.

Those eleven outputs, which the same-family judge passed and the cross-family judge failed, have a family resemblance when you read them together. Ten of the eleven open by restating the question, nine use the phrase "based on the provided context" or a close variant, and eight present a two-item bulleted list where the content would have fitted in a sentence. Those are house-style markers of X's post-training, and the same-family judge treated them as evidence of diligence — several of its rationales cite the answer's structure as a reason the criterion is satisfied, which is not something any of the five criteria mention.

There is a direct test of that reading, and it takes an afternoon. Take 20 answers, have a third model rewrite each one into a different register with the propositional content held fixed — same claims, same citations, no restatement, no bullets, plainer phrasing — and re-judge both versions. The same-family judge passed 17 of the originals and 12 of the rewrites; the cross-family judge passed 14 and 13. A judge whose verdict moves by five examples out of twenty when only the register changes is measuring register, and the criteria say nothing about register.

The four numbers in that probe have to be read as two differences rather than as four levels, and the reason is that the rewriting step is not neutral. A third model rewriting twenty answers will occasionally damage one — drop a hedge that was carrying a condition, or compress two claims into one — so some of the drop from 17 to 12 might be real quality loss rather than register preference. The cross-family judge is the control for exactly that. It reads the same twenty originals and the same twenty rewrites, so any genuine damage is in front of it too, and it moves by one. Comparing the two movements — five against one — is what isolates the part that is about register, because the only thing that differs between the two comparisons is which family is grading. Running the probe on one judge alone would have produced a five-example drop and no way to say what caused it.

Twenty answers is a small probe and the difference between a drop of five and a drop of one is not a precise quantity; what the design buys is a direction that cannot be explained by damage done during the rewrite. The size of the effect still comes from the decomposition above, and this probe is what says the seven points in it are about form rather than about anything the criteria mention.

Inter-annotator agreement and the size of the sample#

Blinded human labels are the reference here, and they are not truth. Three agents labelling 60 outputs will disagree with each other, and if their pairwise agreement is 78% then the 71% figure carries a margin of several points on its own. A gap of 23 points is far outside that margin, and a gap of 4 points would not be. Before treating a human-versus-judge delta as a finding, compute inter-annotator agreement — how often the labellers reach the same verdict as one another on the same output — on the same set, because a judge cannot be shown to disagree with a standard that does not internally agree.

The sample size deserves the same scepticism. Sixty examples give a 95% confidence interval — the band the true rate plausibly sits in, given a sample this small — of roughly ±11 points on a proportion near 75%, which is wide enough that the 87-versus-85 comparison in that table is not a distinction anyone should defend. The decomposition is legible because the large slices are large; the small slices are noise wearing a number.

That ±11 is worth deriving once, because the same three-step calculation prices every sample in this topic. The spread of a proportion comes from multiplying the pass rate by the fail rate, so at three quarters passing that is three quarters times one quarter, a little under two tenths. Divide by the sample size, 60, and take the square root: the result is a bit over five points, and that is one standard error. The conventional 95% band is close to two standard errors either side, which is the ±11. Two consequences follow directly. Halving the band means quadrupling the sample, because the sample size sits under a square root, so four times sixty outputs would be needed to bring it near five points either side. And the band is at its widest near a half and narrows toward the ends of the scale, which is why a judge sitting at 94% is measured more precisely than the humans at 71% on the same 60 outputs.

Nothing here says a same-family judge is unusable. It says the judge's leniency toward its sibling is a systematic bias of a size worth measuring, and that a team gating releases on a same-family judge is running a gate with a known offset. If the suite is comparing this quarter's system against last quarter's under the same judge, a constant offset partly cancels and the trend is still informative; if the suite is answering "is 94% good enough to ship", it does not cancel at all.

A judge-versus-human gap, then, is not a finding — it is a quantity waiting to be decomposed, and the decomposition is cheaper than any of the repairs it points at. Four instruments over one frozen set turns an alarming number into four attributable ones, which is the difference between scheduling a month of rubric work because it was the loudest suggestion in the room and scheduling it because it is nine of the twenty-three points.

The other thing the family swap does not do is make two models independent. Seven points of the gap closed when the judge changed vendor, and the remaining agreement between the two judges is not all attributable to both being right. Both were pretrained on much of the same public web, both may have absorbed the same benchmark data, and a factual claim that neither family knows to be wrong will be marked correct by both with equal confidence. Knowing which half of the problem the swap actually buys is what makes it possible to price the swap.