Reading a score histogram

A judge that returns a score from 1 to 5 produces a distribution, and the distribution is the part worth looking at. A mean collapses it into one number that can only be compared with other means; the histogram underneath says where the judge is confident, where it is not, and which levels of the scale it treats as real. When 71% of a run lands on two adjacent levels, almost all the variance in the metric lives in the two buckets the rubric describes least well, and every movement in the quarterly mean is movement between them.

It is worth stating plainly what produces that number before reading anything into it. An LLM-as-a-judge — a large language model, the same kind of system that writes the answers, prompted instead to grade them — exists because the thing being graded is open-ended prose: a support answer has no single correct string, so the assertions a unit test would make do not apply, and two answers can share almost no wording and both be right. What it grades against is a rubric, the written instructions carried in the judge's prompt that name the properties to look for and the form of verdict to return. The judge design rules of this topic are the choices made around that rubric and that call: whether a verdict is a score or a yes-or-no, how many properties go into a single call, what the model writes before it commits to an answer, which model does the judging, and what the arrangement costs per run. You meet the output of those choices wherever a number stands in for a person reading the text — a quality tile on a dashboard, a check that has to go green before a pull request merges, a threshold that holds a release back.

Seventy-one percent in two buckets#

Here is the distribution behind one such tile: 600 support-assistant answers per quarter, each sent to a judge with the answer, the retrieved context, and a rubric that defines a five-point scale.

A quarterly mean of 3.7 over that distribution can move 0.3 by having 180 answers shift from 3 to 4 and nothing else changing anywhere in the suite. That claim is worth checking rather than accepting, because the arithmetic is what makes the rest of the chapter land. A mean over 600 scores is the sum of all 600 divided by 600. Moving one answer from a 3 to a 4 adds exactly one point to that sum, and one point divided by 600 is one six-hundredth of a point on the mean. Move 180 answers and the sum gains 180, which is three tenths of 600, so the mean gains three tenths. Nothing in that calculation cares which 180 answers moved or why; any 180 will do, and so will ninety answers moving two levels each.

To say what those 180 answers gained, you would need the scale to tell you what separates the two levels, and it cannot: a 4 is not a 3 plus a named property, it is a 3 plus a judge's sense that this one was a bit better. The five-point scale is tempting precisely because it looks like it carries more information than a yes or no, and it does produce a mean, a trend line and a tidy tile. What it does not produce is a quantity whose changes decompose into causes.

The reason those two levels dominate the metric follows from the same division. An answer sitting on a 3 has almost nowhere to go except a 2 or a 4, and 227 of the 600 are sitting there; another 199 are on the 4. With 71% of the mass on two adjacent levels, most of the score-points available to move are held by answers whose only cheap move is into the level next door. The 1s and the 5s are a fifth of the run between them, and an answer at the top of the scale cannot move up at all. So the mean is, in practice, a report on traffic between 3 and 4, whatever the five levels suggest about its range.

What the rubric says about a 3#

Pull up the rubric text that produced those numbers and read the anchors, which is where the shape of the histogram comes from:

Rate the assistant's answer from 1 to 5.

5 — Excellent. Fully answers the question, grounded entirely in the
    provided context, cites the source passage, no unsupported claims.
4 — Good. Answers the question with minor gaps or an uncited claim.
3 — Adequate. Partially answers the question.
2 — Poor. Largely fails to answer or contains unsupported claims.
1 — Unacceptable. Wrong, harmful, or entirely ungrounded.

The 5 is a conjunction of four testable properties: answers the question, grounded in context, cites the passage, no unsupported claims. The 1 is testable in the same way. The 3 is defined by the word "partially", which does not name a property that an answer either has or does not have, and the 4 is defined as a 5 with unspecified "minor gaps". Those two anchors describe a degree of something rather than the presence of something, and the judge has to decide the degree with no threshold to decide it against.

When the judge reaches the point of emitting the score token, it has a distribution over the strings 1 through 5. For an answer that plainly cites the right passage and answers the question, the anchor for 5 matches and that distribution is sharp. For an answer that answers the question but paraphrases the source without quoting it, no anchor matches, and the mass spreads across 3 and 4 in proportions set by whatever the model absorbed about how five-point scales are used in its training data — which is dominated by product reviews, survey instruments and rating datasets where the middle is where raters put things they are unsure about. The judge is not measuring the answer at that point. It is reproducing the prior of the scale.

The chain under that sentence is worth walking, because it explains why one anchor produces a reading of the answer and another produces a guess. A score token is drawn from a distribution the model computes over everything sitting in its context, which here is the rubric and the answer. An anchor written as a conjunction of testable properties puts four questions into that context whose answers are physically present in the text being graded: is the question addressed, is every claim supported by the retrieved passages, is the passage cited, is anything unsupported. Evidence bearing on the choice between 5 and 4 is therefore available, and the distribution comes out sharp for the same reason any well-evidenced continuation comes out sharp. An anchor written as "partially" puts no such question in. Nothing in the answer decides whether a partial answer is partial enough to count as a 3, because the rubric never said how much is enough, and no amount of re-reading the answer can supply a threshold the rubric withheld. With no evidence in the context bearing on the choice, the only thing left to shape the distribution is the model's general expectation about five-point scales — and that expectation is the prior above.

Rubrics acquire this shape honestly. Somebody sits down to write one, finds that describing an excellent answer and a disastrous answer is straightforward because both are recognisable, and then has three levels left to fill in between two things they have already described. The filler words that go into those levels — partially, largely, minor, mostly — are the vocabulary of interpolation, and they are what a person writes when the scale has more resolution than their understanding of the failure modes. The rubric is not sloppy so much as over-specified relative to what anyone can articulate about the middle of the range.

Survey methodologists named this decades before anyone put a language model in a grading loop. A Likert scale is the numbered strongly-agree-to-strongly- disagree instrument that questionnaires are built out of, and central tendency bias is the documented tendency of the people filling one in to avoid its endpoints. It is one of the standard reasons psychometricians insist on behaviourally anchored rating scales, where each point is tied to an observable behaviour rather than to an adverb. The judge inherits the pathology from its training data and from the same underlying cause: an unanchored middle gives a rater nowhere to stand.

The parallel is close enough to be useful and it is not an identity, so it is worth saying where it stops. A person avoids the endpoints partly to avoid committing to a strong claim they might have to defend, and a model grading a support answer has no such stake — nothing follows for it from being wrong. What transfers is the other half of the survey finding, which is that an item whose middle options are described in adverbs rather than in behaviours produces a central pile from any rater at all. The model reaches the middle by a different route, having read a great deal of text in which middles are where uncertainty was recorded, and arrives in the same place.

Re-running the same 600 shows how much of the score is instrument#

The way to test whether the middle is measurement or hedge is to change something that cannot affect quality and see what moves. Take the same 600 stored outputs, re-run the judge at temperature 0, and change only the order in which the four properties are listed inside the rubric text — the same properties, the same scale, one permutation of the sentence order.

import collections

flips = collections.Counter()
for ex in examples:                      # 600 stored outputs
    a = judge(ex, rubric=RUBRIC_ORDER_A)  # score 1..5
    b = judge(ex, rubric=RUBRIC_ORDER_B)
    flips[(a, b)] += 1

moved = sum(n for (a, b), n in flips.items() if a != b)
print(moved / 600)

Two properties of that design are what make the result readable, and both are easy to lose by accident. The outputs are stored rather than regenerated, so the text being graded is byte-identical across the two runs and no difference in the scores can be a difference in the answers. And the perturbation is a permutation of sentence order inside the rubric, which changes no requirement and adds no criterion, so a score that moves under it has moved for a reason that is not about quality by construction. Temperature 0 removes a third source of movement by making the judge's own sampling repeatable, which matters because otherwise every flip would have two candidate explanations and the run would settle neither.

Across the 600, 14.2% of scores changed, and 78 of the 85 changes were between 3 and 4. Scores of 1 and 5 moved on 7 outputs in total. Those counts reconcile directly: 14.2% of 600 is 85 scores, and 85 changes with 78 of them inside the middle pair leaves 7 for the whole of the rest of the scale. The rubric's endpoints are stable under a perturbation that has nothing to do with the answers, and its middle is not, which tells you where the instrument's own noise lives.

Reading the rationales attached to those 85 flipped scores is more informative than the count. In the run where "cites the source passage" appeared first in the rubric text, the rationales for the flipped outputs mention citation in 61 of 85 cases and conclude with a sentence about the answer being incomplete; in the run where it appeared last, 24 of them mention it at all. The judge was weighing whichever property it had read most recently, which is a property of the prompt rather than of the answer, and the score absorbed that weighting silently because a single number has nowhere to record which property drove it.

The same thing shows up when two judges grade the same outputs. Running a second model over the identical 600 gives exact-score agreement of 52% and within-one-point agreement of 91%; quadratic weighted kappa — an agreement statistic that discounts the agreement two graders would have reached by chance alone, and counts a two-point disagreement more heavily than a one-point one — comes out at 0.31, which is the range conventionally read as fair-to-poor. Both judges agree almost perfectly on which answers are terrible and which are excellent, and they disagree constantly about whether a middling answer is a 3 or a 4. Two graders can share a rubric and still hold different internal thresholds for "partially", because nothing in the rubric fixes that threshold.

A 52% match rate falling to a kappa in the low thirties looks like a heavy discount until you do the subtraction, and the subtraction is the part worth learning, because a middle-heavy histogram is exactly the shape that makes it large. Chance agreement is what two graders would achieve if each kept its own habit of using the scale and then answered at random. For two graders with the share column above, the chance of both landing on a 3 is 38% of the time times 38% of the time, the chance of both landing on a 4 is 33% times 33%, and the same product is taken for the other three levels. Adding the five products gives a little under three tenths, so getting on for one match in three is free. Subtract that from the observed 52%, divide by the room that was left above chance, and the result lands close to the 0.31 quoted — the quadratic weighting shifts it a little, because it forgives the many one-point disagreements and punishes the few two-point ones, but the size of the discount is set by the pile in the middle. Two judges that both hedge toward 3 and 4 will agree often without either of them having read anything.

Set those numbers against a quarterly drift of 0.3. The standard deviation of the score distribution is about 0.95, which is what you get by taking each level's distance from the mean, squaring it, weighting it by the count in that bucket, averaging, and taking the square root; the standard error of a 600-sample mean is that 0.95 divided by the square root of 600, and the square root of 600 is a shade under twenty-five, so the standard error comes to 0.039. A 0.3 movement is therefore many standard errors and clears statistical significance easily.

It clears nothing else, and the reason is that the standard error answers a question nobody in the review was asking. What it prices is the luck of the draw: if you had sampled a different 600 answers from the same population and graded them with the same instrument, this is how far the mean would wander. It says nothing whatever about what happens when you hold the 600 answers fixed and disturb the instrument, which is the thing that actually differed between two quarters in which the rubric was edited and the judge model was upgraded. Those disturbances have their own numbers, and they are larger: a permutation of the rubric's sentence order moved the mean of the same unchanged outputs by 0.08, and swapping the judge model moved it by 0.24. A quarterly change of 0.3 is inside the range that the measurement apparatus produces on its own when you jiggle it.

The 0.08 is smaller than the flip count makes it sound, and the gap between the two is informative rather than reassuring. Eighty-five individual scores moved, and yet the mean moved by 0.08, which over 600 answers is a net gain of about fifty score-points. Eighty-five flips producing a net of fifty means roughly two-thirds of them went up and a third came back down; the movements largely cancel, and what survives is a small residual drift. An experienced reader of dashboards will note that the tile shows only the residual. The 85 answers whose recorded quality changed under a rewording are invisible in it, and they are the answers an engineer would be sent to look at.

Three shapes and what each says#

Once you know the middle is where the criteria run out, the histogram becomes a diagnostic for the rubric rather than a report on the system. Three shapes come up repeatedly and mean different things.

Only the third shape supports a mean. A tile that reads 3.7 from the first shape is reporting the position of a hedge, and if the middle buckets are noise then a mean computed over them is a weighted average of noise and signal with no way to separate the two afterwards.

There is an obvious repair to reach for at this point, which is to write a better anchor for the 3. It does help, and it helps in a way worth noticing: a 3 that reads "answers the question but does not cite a source passage" is stable across re-runs, because it names a property the judge can check. Having written it, though, you now have a criterion — "cites a source passage" — that either holds or does not, and you have expressed it as a position on a scale rather than asking about it directly. Every anchor that makes a scale point reliable does so by converting it into a binary test, which is the argument for skipping the scale.

The belief that repair is fighting is worth stating outright, because it is reasonable and it is wrong. Five levels look like more information than two: a yes-or-no throws away everything about how good the answer was, while a 1-to-5 keeps a gradation, and one number that can take five values obviously carries more than one that can take two. The step that fails is the last one. A level carries information only when something decides which level an answer belongs on, and the deciding has to be repeatable by a different reader, or a different run, or the same judge with the rubric's sentences in a different order. Levels 3 and 4 have no such decider, so the extra resolution is resolution the instrument cannot deliver. A binary criterion has less resolution and all of it is real; the five-point scale offers more and cannot supply the part that matters, which is why the re-run above changes 85 scores on the middle levels and 7 everywhere else.

A middle-heavy histogram says nothing about the product#

The diagnosis is evidence about the rubric and not evidence about the system being graded. A support assistant with a middle-spiked histogram may well be mediocre; these numbers say nothing either way, because the instrument that produced them cannot resolve mediocre from good. Teams sometimes read the diagnosis backwards and conclude their quality is fine and the eval is broken, which is the same mistake in the other direction.

The failure is also not exclusive to five-point scales as a format. A 1-to-10 scale has the identical problem with more buckets to hide it in — the mass moves to 6, 7 and 8 — and a three-point scale of good/acceptable/bad has it in a single bucket. What matters is whether each level names a property that can be checked, and scales invite levels that name degrees.

Nor does binary conversion make disagreement disappear. Two judges that disagreed about 3-versus-4 will disagree about pass-versus-fail on the criterion underneath, and the disagreement rate may not drop much at all. What changes is that the disagreement becomes attributable to a named requirement, so you can read the two rationales and decide which one is right, whereas nothing productive follows from one judge saying 3 and another saying 4.

A histogram, then, is the cheapest reading available on a judge, and it is readable before any calibration exercise, any human labelling and any argument about model choice. Mass concentrated on the levels a rubric defines with adverbs tells you that those levels are where the judge puts what it cannot decide, and that the mean you have been trending is a weighted average over a hedge. The confirming experiments are two re-runs over stored outputs and cost the price of one suite run each.

The rest of this topic is about what the judge harness looks like once the scale is gone. Replacing five levels with pass or fail on each named criterion changes what a dashboard can tell you, and it takes away one thing it could do before, which is the subject of the next chapter.