Rubric reliability

A rubric is the written standard an output is graded against — the conditions an answer has to satisfy, phrased in the words a grader applies them by. It is the same artefact whether a person or a model does the applying, and that is the property this chapter turns into a test: a rubric can be handed to two people and measured before any of it is automated. The measurement takes a morning, costs nothing but two people's attention, and returns a number about the rubric rather than about the system being graded.

It is worth being explicit about what the rubric is about to become, because everything the measurement can tell you depends on it. An LLM-as-judge is a large language model — an LLM, the same class of model that produced the answers in the first place — handed someone else's output together with a written standard, and asked to return a verdict on it: a pass or a fail, a score out of five, or a choice between two candidates. It exists because the property being checked is prose, and prose has no assertion you can run against it. A documentation assistant emits a few thousand answers a week, "did this answer help the user" has no correct string to compare with, and two engineers grading forty answers by hand spend most of a morning on it. A model doing the reading is what turns evaluation into something that can run on every commit instead of once a quarter. You meet the pair wherever free-text quality has to be measured at volume: an eval suite that gates a release, a nightly dashboard carrying a quality score, a filter that drops weak examples from a fine-tuning set, a guardrail that inspects an answer before a customer sees it. In each of those the model supplies the reading and the rubric supplies the meaning of the verdict.

Sixty percent is worse than it reads#

Here is the measurement, run on forty answers from a documentation assistant by two engineers on the same team, working independently from one instruction:

Rate the answer's helpfulness from 1 to 5. 5 is excellent, 3 is adequately helpful, 1 is not helpful.

They matched on twenty-four. Every one of the forty answers was graded by both engineers, so every answer is one opportunity to agree or disagree and forty is the denominator: twenty-four out of forty is three in every five, which is the sixty percent. Sixty percent raw agreement sounds mediocre rather than alarming, and the temptation is to treat the sixteen mismatches as the ordinary friction of human taste and move on to the automation, where at least the scoring will be uniform. That reading has the causality backwards, and the first correction to make is that sixty percent is not sixty percent.

Raw agreement flatters any rubric whose scores pile up in the middle, and this one does. Graders using a 1-to-5 helpfulness scale on outputs from a system that mostly works spend nearly all their time choosing between 3 and 4. In this sample the combined distribution was 5% ones, 10% twos, 35% threes, 35% fours and 15% fives. Two graders drawing independently from that distribution — agreeing by coincidence rather than by shared understanding — would match on 0.05² + 0.10² + 0.35² + 0.35² + 0.15², which is 28%. Cohen's kappa corrects for exactly that, giving (0.60 − 0.28) / (1 − 0.28) = 0.44. On the conventional reading of kappa that is moderate agreement, the band where two people are clearly responding to something real and clearly not responding to the same thing.

That sum of squares is short enough to hide two assumptions and one piece of arithmetic, and all three are worth having in hand, because the same move underlies every chance-corrected figure in this topic. Take one answer and suppose neither grader is responding to it at all — each simply reaches for a score with the frequency they use that score overall. They agree on this answer only if both reach for the same one. The chance that both reach for a 3 is the chance the first does, times the chance the second does, and multiplying the two is permitted exactly because they are being assumed independent: independence is the statement that knowing one grader's score tells you nothing about the other's, which is what "by coincidence" means. Here one combined distribution stands in for both graders, so each of those products is a share multiplied by itself, and that is where the squares come from. It is an assumption rather than a measurement — it holds only if the two engineers used the five scores in similar proportions, which they roughly did.

The five squares are then added rather than combined some other way because the five ways of agreeing are mutually exclusive: two graders cannot both land on 3 and both land on 4 on the same answer. Reading the sizes off, each 35% row squares to a little under an eighth, so the two middle scores alone account for just under a quarter of the total, and the 15%, 10% and 5% rows contribute the remaining few points between them. Squaring always shrinks a share, because every share is below one, and it shrinks the small ones far harder than the large ones. That is the mechanism under the claim that raw agreement flatters a rubric whose scores pile up in the middle: the more a rubric's marks concentrate onto a few values, the more agreement two graders are handed for nothing. Had all five scores been used equally often, each share would be a fifth and the five squared fifths would sum to a fifth — twenty percent of free agreement instead of twenty-eight.

The correction itself is one subtraction and one division, and the division is the part that is easy to accept without understanding. Of the sixty points of observed agreement, twenty-eight would have arrived even if neither grader had read anything, so the agreement that had to be earned is sixty minus twenty-eight, which is thirty-two points. Those thirty-two points are not thirty-two out of a hundred, though. The ceiling is a hundred percent and twenty-eight of it was free, so the room available above coincidence is a hundred minus twenty-eight, which is seventy-two points. Kappa is the share of that room actually taken: thirty-two over seventy-two, which is 0.44. Dividing by the headroom rather than by a hundred is what makes the figure comparable across rubrics whose score distributions differ, since a raw excess of thirty-two points means something quite different when only a little room was available than when nearly all of it was.

It is worth being honest that this rescaling has a cost, and it is the reason kappa reads as harsh. The number is a share of headroom rather than a share of answers, so it cannot be quoted as "the graders agreed on 0.44 of the sheet" and nobody should try. The extreme case shows why the distinction is real rather than pedantic: if both graders had given every answer the same score, coincidence alone would predict complete agreement, the headroom would be nothing at all, and the division would have no meaning. A rubric that concentrates its marks is therefore not merely flattered by raw agreement — it approaches a state in which raw agreement carries no information about the rubric at all.

The shape of the disagreements says more than the headline. Splitting the sixteen mismatches by size and by what the answers had in common produced this:

The twenty-four agreements were the easy cases: answers that named the right setting and stopped, and answers that were plainly off-topic. Neither grader needed a rubric for those. Every case where the rubric was supposed to earn its keep is in the other rows.

The four large gaps are the most instructive. All four were fluent, structured answers containing one incorrect claim — a menu path that does not exist in the current version of the product. One grader scored them 2, reasoning that an answer which sends a user to a nonexistent screen has failed at the only thing that mattered. The other scored them 4, reasoning that the answer was well-organised, covered the concept correctly, and had one fixable error. Both readings are defensible under "rate helpfulness from 1 to 5", which is the problem. The scale never said whether a fabricated navigation step is a deduction or a disqualification, so each grader supplied their own answer and the sheet recorded the difference as though it were signal about the answers.

The move from "both readings are defensible" to "the fault is in the rubric" is the step this whole chapter rests on, and it is worth walking rather than asserting. A grader is careless when their score cannot be justified from the rubric they were given. Each of these two scores can be: the instruction asks for helpfulness on a five-point scale, one grader read a nonexistent menu path as a failure at the only thing the answer was for, the other read it as one fixable error inside an otherwise sound explanation, and the instruction contains no sentence that rules against either. Since neither reading can be shown to violate the standard, the difference between them is not located in the people applying it. It is located in the standard, which admits both. Underdetermination is the name for that property: a rubric is underdetermined when the text of it is consistent with more than one verdict on the same output.

The reason this is hard to catch is that underdetermination is invisible from inside. Whoever wrote the instruction reads it with the cases they had in mind already loaded, so every phrase resolves as they intended and the document seems plainly clear. Re-reading it more carefully does not help, because the second reading is done by the same person with the same cases loaded. The belief a reader is most likely to carry out of a rubric review — this is clearly written, so two graders will apply it the same way — is therefore not a lazy belief. It is the only belief available from the author's position, and the sole evidence against it is two people who did not write the rubric applying it independently and coming back with different sheets. That is what makes the exercise an instrument rather than a formality.

Conditions a grader can check, not a quality a grader can feel#

The repair is to stop asking for a judgment about a latent quality and start asking for verdicts on conditions that are present or absent in the text. The mechanism behind that repair is worth stating before the criteria themselves, because it explains why a more carefully worded five-point scale would not have worked. "Helpfulness" is a quality a reader attributes to an answer, not a feature the answer either carries or lacks, so there is no place in the text at which a grader can point to settle it. Two people can only agree on a boundary both of them can find, and a boundary that exists only in each grader's own reaction is not a boundary either of them can find. Rewriting the description of a latent quality more precisely leaves it a description of a latent quality. Replacing it with a question about the text moves the boundary into the artefact both graders are looking at, which is the one place a disagreement can be settled by rereading rather than by negotiation.

What a rubric does to a decision, put plainly, is break one open judgment into a sequence of narrower ones. Reliability is then a property of how narrow each of those got — how little room a grader has to supply their own ruling — rather than a property of how well the document reads. That is why the repair below is measured in what each criterion asks about rather than in the quality of its prose. For the documentation assistant, the three that carried nearly all the disagreement became:

criteria:
  - id: names_the_setting
    question: >
      Does the answer name the specific setting, field or command the
      user asked about, using the exact label it has in the product?
    pass_example: >
      "Turn on Settings > Security > Enforce SSO for the workspace."
    fail_example: >
      "You can enable single sign-on from your security settings."
    boundary: >
      Naming the correct setting with slightly stale capitalisation is a
      PASS. Naming the correct area without the setting is a FAIL.

  - id: states_the_limitation
    question: >
      If the requested behaviour is unavailable, restricted to a plan, or
      only available under a condition, does the answer say so explicitly?
    pass_example: >
      "Enforced SSO is available on Enterprise plans only."
    fail_example: >
      "Enable it under Security." (Requested behaviour is Enterprise-only
      and the answer does not mention it.)
    boundary: >
      A correct answer to a question with no limitation is a PASS by
      vacuity. Hedging with "may not be available on all plans" when the
      limitation is known and specific is a FAIL.

  - id: no_invented_navigation
    question: >
      Is every menu path, button label and CLI flag in the answer one that
      exists in the documented product surface?
    pass_example: >
      "Run `acme workspace sso enable --enforce`."
    fail_example: >
      "Open Admin > Identity > SSO Enforcement." (No Identity menu exists.)
    boundary: >
      Describing a step without naming a path is a PASS. Naming a path
      that existed in a previous major version is a FAIL.

Three things changed, and each of them did work. Every criterion is a question about the text rather than about the reader's reaction to it, so a grader who dislikes the answer's tone has nowhere to put that dislike. Every criterion carries a fail example that shows the near-miss rather than the obvious failure, because the near-miss is where graders diverge and the obvious failure is where they never did. And the boundary note names the specific ambiguity a grader will actually hit, which is a different thing from restating the criterion in other words. "Naming the correct area without the setting is a FAIL" is a ruling; "be specific" is a hope.

Re-running the same forty answers with the new sheet produced agreement on thirty-six. The four survivors clustered on states_the_limitation, where both graders had to decide whether a plan restriction the user had not asked about still counted as one they had to mention. The rubric had not said, so a sentence was added ruling that limitations count only when they block the action the user described. That is the loop worth internalising: the disagreements are a queue of underspecified cases, and each one you resolve is a decision that would otherwise have been made silently and differently on every future run.

The switch from a five-point scale to binary conditions is doing part of the work here, independently of the wording. A 1-to-5 scale has four boundaries to agree on and the two middle ones carry no shared definition — nobody can say what separates a 3 from a 4 in general, which is why nine of the sixteen original disagreements lived exactly there. Three binary criteria have three boundaries, each attached to a specific observable. That does not make the judgment objective; it makes each judgment narrow enough that a boundary example can settle it.

That switch carries a cost the raw numbers conceal, and it should be said before thirty-six of forty is read as a triumph. Everything in the chance-agreement calculation above turns on how many values the graders were choosing between: five values used in similar proportions leave a fifth of the agreement to coincidence, and two values used in similar proportions leave half of it. Fewer categories means a higher floor, and a higher floor means a larger raw number for the same quality of rubric. The second round's raw agreement is therefore not comparable with the first round's — a sheet that moved from a five-point scale to binary conditions would report some improvement even if nothing about its clarity had changed. The comparison that survives the change is the chance-corrected one, and computing it for the second round needs the rates at which each grader passed and failed on the new sheet, which the exercise has already produced and which nobody usually bothers to write down.

Narrow criteria still have to roll up into something, and how they roll up is a separate decision that also has to be made in advance. Requiring all three to pass gives a pass rate of 55% on these forty answers; requiring at least two gives 82%. Both are defensible readings of "a good documentation answer" and they describe the same system, twenty-seven points apart. Weighting is available too — a fabricated menu path is arguably disqualifying in a way that a missing plan caveat is not — but a weighted score reintroduces exactly the scale problem the criteria were meant to remove, so the usual answer is to keep the roll-up as a rule about which criteria are mandatory rather than as arithmetic.

The ordering of those two pass rates is fixed before any answer is graded, and seeing why makes the choice between them legible. An answer that passes all three criteria has, in particular, passed at least two of them, so the answers admitted by the strict rule sit entirely inside the answers admitted by the loose one. A subset cannot be larger than the set containing it, which is why requiring all three can never report a higher pass rate than requiring two — on any sample, with any criteria, without measuring anything. The gap between the two figures is therefore not noise and not a difference of opinion about quality. It is exactly the share of answers that passed two criteria and failed one, and those twenty-seven points are the entire population the roll-up decision is about. Choosing a rule means deciding what should happen to an answer that satisfies most of the standard and misses one part of it, which is a question somebody in the room can answer. Which of the two pass rates is the correct pass rate is not.

How to run the disagreement measurement#

The exercise that produced the 24-out-of-40 number is worth specifying, because most of its value comes from procedural details that are easy to lose.

The two graders work independently and do not discuss the rubric first. A five-minute alignment chat before grading is the single most common way this measurement gets destroyed: it produces shared conventions that live in the graders' heads, agreement rises, and nothing has been written down that a judge could apply. The measurement is of the rubric as written, so the graders have to work from the rubric as written.

The items are shuffled and the source is hidden, so neither grader can anchor on the order or on which system produced what. Forty is a reasonable size: the standard error — the typical distance between a figure measured on a sample and the figure you would get from the whole population — on an agreement proportion around 0.6 at n=40 is about 7.7 points, which is enough to distinguish 60% agreement from 85% and not enough to distinguish 80% from 85%. If you need the finer distinction you need a few hundred items, and you almost never need it — the decision this number feeds is "is the rubric usable yet", which is a coarse question.

Where the 7.7 comes from is worth having, because the shape of the calculation is what explains why forty items settle a coarse question and no finer one. The quantity is the proportion multiplied by its complement, divided by the number of items, square-rooted. Six-tenths times four-tenths is a little under a quarter; a quarter spread across forty items is a few thousandths; and the square root of a few thousandths is a shade under eight hundredths, which is the 7.7 points quoted. Two features of that expression do all the work. The item count sits under the division and the whole thing sits under a square root, so precision is bought with items and bought inefficiently — halving the interval costs four times the labelling and quartering it costs sixteen times. And the numerator is largest when the proportion is near a half, which means an agreement figure in the middle of the range is the most expensive kind to pin down, and a rubric under review is exactly where that figure sits.

Reading the interval back onto the decision is one comparison. Two figures are distinguishable when the gap between them is large against the uncertainty each carries. The gap between 60% and 85% is twenty-five points, more than three times 7.7, so no plausible amount of sampling luck closes it. The gap between 80% and 85% is five points, comfortably inside a single 7.7, so a sheet reading 80% and a sheet reading 85% are the same sheet as far as forty items can tell. A team that re-ran the exercise after a rubric edit and celebrated a move from 80% to 85% would be celebrating the sample rather than the edit.

Disagreements are logged rather than resolved by majority. A third grader breaking ties produces a labelled set of items and destroys the diagnostic, because the point is not to obtain labels for these forty items but to find the places where the rubric admits two readings. The joint session afterwards goes through each disagreement and produces one of two artefacts: an edit to a criterion's wording, or a ruling added to its boundary note. Both go into the rubric file. Neither is a verbal agreement.

Two rounds is typical. If agreement is still under 85% after the second round, the usual cause is a criterion that is genuinely about taste — "is the answer well organised" — which should be deleted rather than sharpened, because no wording will make it reproducible and its presence contaminates the criteria around it.

Consistency is cheap and proves nothing about validity#

The objection that arrives at this point is that human disagreement is beside the point, because the judge will be run at temperature 0 — the sampling setting that has the model take its highest-probability continuation every time, rather than sampling among the plausible ones — against a fixed prompt, and will therefore be more self-consistent than either engineer. The premise is true and the conclusion does not follow, because reliability and validity are separate properties and only one of them is being improved.

Reliability is agreement between measurements of the same thing. Validity is whether the measurement corresponds to the property you named. A judge running the original 1-to-5 scale at temperature 0 has excellent reliability: score the same answer a hundred times and you will get the same number nearly every time. What it does not have is any claim on the meaning of that number, because the rubric that produced it was one that two informed people read two different ways. The judge resolved the ambiguity too — every model does, it has to emit something — but it resolved it by whatever its own priors suggested, once, invisibly, and then applied that resolution uniformly forever.

That the two properties come apart is easy to accept as a slogan and worth tracing as a mechanism, because the tracing is what shows the gap cannot be closed by a better setting. Once the prompt, the rubric and the model are fixed, a judge has exactly one remaining source of run-to-run difference: the step that samples among plausible continuations. Temperature 0 deletes that step, so two runs over the same answer take the same path and return the same verdict, and reliability rises to nearly one by construction. What the deletion does not touch is the mapping from the rubric's words to a score. That mapping lives in the model's weights, the rubric's ambiguity is resolved inside it, and removing the noise around a resolution cannot inspect the resolution. The setting makes the answer repeatable without making it answerable.

The result is a metric with low variance, a smooth trend line and no interpretation. It looks better on a dashboard than a noisy valid metric, and that is the trap: variance is visible and invalidity is not. A team watching a quality score sit at 4.1 ± 0.05 week over week has every surface signal of a healthy instrument. The failure only surfaces when someone asks what a 4 means, and two people in the room answer differently, and neither of them can be shown to be wrong. The strong version of the claim is that a reproducible number nobody can define is worse than no number, because it consumes the attention that would otherwise go to reading outputs. A missing metric leaves a gap that people fill by looking at the product; a confident, stable, meaningless metric fills the gap and stops them looking.

Rubric criteria are for the conditions that need a reader#

Sharpening a rubric fixes one specific failure and it is worth being exact about which. It does not address whether the sample being graded is the right sample — that work belongs to a golden set of curated inputs, and the criteria above assume the forty answers came from one. It does not turn a subjective property into an objective one; states_the_limitation still requires a reader to decide what counts as blocking the user's action, and the boundary note narrows that judgment rather than eliminating it.

Any criterion that could be decided with no judgment at all should not be in a rubric, because a string match or a schema validation will check it faster, cheaper and with no possibility of drift. That is the line worth holding when a rubric starts to grow: rubric criteria are for the conditions that need a reader, and everything else belongs in code.

Nor is a rubric a style guide. It is easy, once you start writing observable conditions, to keep going into formatting, tone and length, and every criterion you add of that kind is a criterion the judge will happily reward. Some of that is legitimate — a documentation answer that buries the setting in paragraph four is worse — but each such criterion has to be defensible as something a user is harmed by, because the judge will treat all criteria as equally load-bearing unless you weight them.

A rubric, then, is measurable before it is automated, and the measurement is two people and a morning. Agreement on 24 of 40 is not a fact about two engineers; it is a fact about a standard that admits two readings, and every reading it admits will be resolved by whatever applies it next. Three observable conditions with boundary rulings brought the same forty answers to 36, and the four that survived became a written ruling rather than a difference of opinion.

The largest thing that does not buy is the judge. Two humans now apply these three criteria the same way, which establishes that the criteria are applicable and says nothing about whether a model applies them. A model reading names_the_setting brings its own inclinations to the reading: a preference for answers that are longer, better organised, more confident in tone, and more similar to what it would have written. Those inclinations do not announce themselves in the score, and they correlate with the rubric often enough that the judge will look correct on most examples. Which means the next thing to build is not the production wiring but the test that tells you whether the judge is scoring these three conditions or scoring the prose around them.