Probe sets for judge bias

A probe set is a small collection of paired items built so that the rubric-correct verdict on each one is known in advance. The two members of a pair carry the same rubric-relevant content and differ in exactly one property the rubric does not mention — how polished the writing is, how long the answer is, whether it hedges. Because you built the pair, you know which member should pass; if the judge disagrees, the disagreement isolates the property you varied. That is the whole design, and it is why a probe set can find a bias that thousands of ordinary graded items cannot.

The isolation is worth taking slowly, because it is the only reason any of this works. Hand a judge a single polished answer with a wrong menu path in it and watch it pass. That verdict is consistent with two different stories: the judge may be scoring the polish, or the answer may be closer to correct than you thought and the judge may be right. Nothing about one verdict separates those, because the only thing available to compare it against is your own reading of the same item. Add a second answer carrying identical rubric-relevant content and the comparison stops being with your reading and starts being with the judge's own other verdict.

Two pair shapes do that isolating, and they do it in slightly different ways. In the first, the rubric-relevant content is held identical and only the non-rubric property moves — the same answer, then the same answer padded with three paragraphs of accurate context. The rubric-correct verdict is therefore the same on both members, so any difference between the judge's two verdicts has to have been caused by the padding, since the padding is the only thing that differs. In the second, the property is set deliberately against the content: a polished answer that is factually wrong, beside a scruffy answer that is right. Here the rubric-correct verdicts differ, and they differ in the direction opposite to the surface, so a judge following the criterion and a judge following the writing give opposite answers on the same pair. The first shape asks whether a property moves the verdict at all. The second asks whether it overpowers the rubric when the two disagree, which is the failure that reaches production.

What neither shape needs is anybody's agreement about what a good answer looks like, and that is what makes a probe diagnostic rather than merely another test. The verdict each item deserves was fixed at the moment the item was written, by the person who typed in the broken menu path or left the plan restriction plainly stated. When the judge's answer differs from the one built in, the difference is therefore information about the judge and not about the item. An ordinary graded item has no such property: what it deserves is a matter of somebody's reading, and a disagreement between a judge and a reader is an argument rather than a measurement.

That is also where the comparison with ordinary grading stops being a question of sample size. On real traffic the non-rubric property and the rubric mostly agree — ordinary items are where every bias agrees with the rubric — so a judge scoring the criteria and a judge scoring the prose hand back nearly the same verdicts across a large sample. No quantity of such items separates the two explanations, because the sample contains almost no case on which they disagree. Constructing the pair is a way of manufacturing exactly those cases, which is why a dozen of them carry information that any number of naturally occurring items does not.

The reason a probe set is needed at all is that a rubric two people apply the same way tells you nothing about how a model applies it. A model brings its own inclinations to a criterion — longer answers, better organised ones, more confident ones, ones more similar to what it would have written — and none of those announce themselves in a verdict.

What the judge is, concretely#

A rubric judge is a model call with a system prompt containing the criteria, a user message containing the item under evaluation, and an output schema that forces a per-criterion verdict rather than a paragraph of appraisal. The schema is where most of the reliability comes from, because a judge asked for prose will produce a considered opinion that has to be parsed, and a judge asked for a structured verdict per criterion has to commit to each one separately. The three criteria below are the documentation-assistant rubric from the previous chapter: whether an answer names the specific setting asked about, whether it states any plan restriction that blocks the user's action, and whether every menu path it gives exists in the product.

import json
from anthropic import Anthropic

client = Anthropic()

CRITERION_IDS = ["names_the_setting", "states_the_limitation", "no_invented_navigation"]

SCHEMA = {
    "type": "object",
    "properties": {
        "criteria": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "id": {"type": "string", "enum": CRITERION_IDS},
                    "verdict": {"type": "string", "enum": ["pass", "fail"]},
                    "evidence": {
                        "type": "string",
                        "description": (
                            "Verbatim span from the answer that decides this "
                            "criterion. Empty string only if the answer contains "
                            "no relevant span."
                        ),
                    },
                },
                "required": ["id", "verdict", "evidence"],
                "additionalProperties": False,
            },
        }
    },
    "required": ["criteria"],
    "additionalProperties": False,
}

JUDGE_SYSTEM = f"""You grade documentation answers against a fixed rubric.

{open("rubric.yaml").read()}

Grade each criterion independently. For each one, quote the span of the
answer that decides it before giving a verdict. Judge only what the rubric
asks about. Length, tone, formatting and organisation are not criteria and
must not influence any verdict."""


def judge(question: str, answer: str) -> dict[str, str]:
    response = client.messages.create(
        model="claude-opus-5",
        max_tokens=2000,
        system=JUDGE_SYSTEM,
        output_config={
            "format": {"type": "json_schema", "schema": SCHEMA},
            "effort": "low",
        },
        messages=[
            {
                "role": "user",
                "content": (
                    f"<question>{question}</question>\n"
                    f"<answer>{answer}</answer>"
                ),
            }
        ],
    )
    verdicts = json.loads(response.content[0].text)["criteria"]
    return {c["id"]: c["verdict"] for c in verdicts}

The evidence field is doing real work and is worth defending against the instinct to drop it for cost. A judge that must quote the span deciding no_invented_navigation has to locate a menu path in the text before ruling on it, and when the ruling is wrong the quote tells you whether the judge misread the answer or misread the rubric. Those are different bugs with different fixes. Without the quote, a wrong verdict is an opaque token and all you can do is stare at it. The field costs perhaps 30 output tokens per criterion, which at Opus pricing of $25 per million output tokens is about $0.0022 per graded item across three criteria — roughly two dollars per thousand items, which is not the line worth trimming.

That figure is worth deriving rather than accepting, because every cost in this chapter is the same three steps. Thirty output tokens on each of three criteria is ninety output tokens for the item, and the criteria can be added like that because the schema gives each one its own evidence string rather than sharing one between them. Twenty-five dollars per million tokens is a rate, and the quickest way to apply a rate that size is to scale it down twice: a million tokens at twenty-five dollars makes a thousand tokens two and a half cents, and ninety tokens is a shade under a tenth of a thousand. A shade under a tenth of two and a half cents is a little over two tenths of one cent, which is the $0.0022 above, and multiplying by a thousand items moves the decimal point three places to reach roughly two dollars. Each step is a multiplication by a constant, which is allowed here only because the evidence field's length does not vary much from item to item — the schema asks for a span rather than an argument, so the per-item figure behaves like a fixed price and not like an average with a long tail.

Two other choices in the judge call above are deliberate. Grading happens at effort: "low" — the dial that controls how much internal reasoning the model does before it commits to an answer — because the criteria are narrow and the judgment is shallow. Extending the reasoning does not give the judge more evidence to work from, since the answer and the rubric are both already in front of it and nothing further arrives during the thinking. What the extra length mostly buys is room to compose a justification, and those tokens are output tokens, which the arithmetic above prices at the expensive rate. A judge that reasons at length about a three-condition rubric is spending tokens on rationalisation. And the sampling parameters that used to be the standard way to pin a judge down, both of which govern how much randomness enters the choice of each next token, are simply absent — temperature and top_p are rejected outright on current Anthropic models, which means judge stability is now something you measure rather than something you configure. Running the same item through the judge five times and checking that the verdicts match is a cheap standing check and belongs in the same test suite as the probes.

That check is not hygiene, and it changes how every probe result below can be read. If the same item can come back with different verdicts on different calls, then one pass over the probe set is a single sample of the judge rather than a reading of it, and a shortfall on that pass could be a stable preference or could be the judge wobbling. The two call for opposite repairs. A stable preference is fixed by changing what the judge looks at, which is what the rest of this chapter does. A wobble is fixed by voting over repeated calls or by removing whatever makes the decision marginal, and no amount of prompt wording touches it. Running the item five times and seeing five identical verdicts is what licenses treating a single probe run as a measurement of the judge.

The four properties worth probing first#

Twelve base items with two variants each is 24 judge calls. At roughly 1,800 input tokens per call — the rubric with its boundary examples is most of that — and 220 output tokens, the whole probe run costs about $0.35 at Opus rates and finishes in under a minute if you fan out. This is the cheapest diagnostic in the topic and the one most often skipped, because a judge that has never been probed looks fine: it produces plausible verdicts on ordinary items, and ordinary items are where every bias agrees with the rubric anyway.

The two halves of that thirty-five cents are worth separating, because which half dominates is the part people get wrong. The output side prices straight from the figures already on the page: 24 calls at 220 output tokens each is a little over five thousand tokens, and at two and a half cents per thousand that is about thirteen cents. Everything remaining in the thirty-five is input. The input side runs past forty thousand tokens across the run, because 1,800 tokens go up on every one of the 24 calls, and the twenty-odd cents left over for that many tokens backs out an input rate several times below the output one. So the input side is the cheaper rate applied to a far larger pile, and the pile wins.

The reason the pile is that large sits in the aside beside the per-call figure: the rubric with its boundary examples is most of the 1,800. A model call carries nothing over from the call before it, so the rubric is re-sent whole and charged again on every one of the 24, though it is the same bytes each time. That is why an estimate made by thinking about what the judge writes comes in low every time. The visible product of a judge call is a short structured verdict, and the invisible bulk is a rubric the judge re-reads from the beginning on every item.

Reading a probe score needs one thing that an ordinary eval score cannot supply: the number an unbiased judge would get. For most measurements that number is unknown, which is why an accuracy figure on its own means little. A probe set is the exception, and the reason is the construction rather than anything about the judge — every item's correct verdict was settled when the item was written. A judge carrying none of the four biases scores 24 of 24, so the ceiling is known before the run rather than estimated after it, and a probe score reads straight off as a shortfall from it. Five verdicts below 24 is five answers the judge got wrong on items whose correct answers were never in dispute. That is why so few items can still be worth acting on: the count is tiny, and the thing it is being compared against is exact rather than estimated. It stops being worth acting on if the stability check above has not been run, because then a shortfall might be five wobbles rather than five preferences.

The denominator is calls rather than pairs, which is worth pinning down before the numbers arrive. Twelve pairs produce 24 judge calls, each call returns its own verdict, and each verdict is scored against the one its item was built to deserve. A run reported as 19 of 24, which is what this judge produced, therefore describes 24 separately graded answers of which five came back against construction — not five bad pairs.

The house-style probe deserves a note, because it is the one that sounds like superstition and is not. A judge tends to score more generously an answer written the way it would have written it — same hedging habits, same structural instincts, same register. When the system being evaluated and the judge are the same model family, that preference is pointed at your own outputs, and it inflates every score in a way no rubric wording addresses. The practical mitigations are to build the house-style probe pair by having a different model rewrite a correct answer in its own voice, and, where the stakes justify it, to judge with a model from a different family than the one under test. Neither eliminates the effect. Measuring it is what lets you decide whether it is large enough to care about.

Model choice for the judge itself is a related decision with a cheap resolution. A smaller, faster model costs perhaps a fifth as much per graded item, which matters on a nightly run over thousands of items, and it will be worse at exactly the borderline cases the rubric exists for. The way to settle it is to run both over the same probe set and the same 40 hand-graded items from the previous chapter, and compare. On this workload the smaller judge scored 21 of 24 probes against the larger judge's 24 and diverged from the human labels on 6 of 40 rather than 2 — a real difference, and one worth paying to avoid on a gating metric while being perfectly acceptable for a coarse triage pass.

Both of those comparisons are small, and it is worth saying plainly why the two together carry weight that neither carries alone. The first denominator is 24 because that is the whole constructed set, and its ceiling is exact, so 21 against the larger judge's 24 is three verdicts the smaller model got wrong on items where the larger model got none. The second denominator is 40 because that is the hand-graded set from the previous chapter, and its ceiling is not exact at all — those labels are somebody's reading — so 6 against 2 is four extra disagreements with a standard that is itself fallible. Three verdicts and four disagreements are counts small enough that a different afternoon could have produced different ones, and either on its own would be a hint. What makes the pair persuasive is that two instruments measuring different things, one against construction and one against people, moved the same way.

The cost side of that decision has an asymmetry in it that is easy to miss. Four fifths of the per-item price is a large saving on a nightly run over thousands of items and an irrelevance on the probe set, where four fifths of thirty-five cents is a saving of pennies on something that runs once per change. So the two models are not competing on the same axis. The cheaper one is buying volume, and the volume it buys is worth having only where the borderline cases the rubric exists for are rare enough not to decide anything.

Fixing the judge rather than the rubric#

The instinct after a probe failure is to add another criterion. That is the wrong repair, because the criteria are not the thing that failed — the three conditions correctly describe what a good answer does, and two humans had already shown they could apply them. What failed is the judge's reading of them, so the fixes belong in the judge prompt and the call. Four changes are worth trying, one at a time, each re-run against the same 24 probes:

Requiring the evidence quote did most of the work, and the reason is mechanical rather than motivational. To rule on no_invented_navigation the judge has to produce the specific path string from the answer, and once that string is sitting in the output as the span under consideration, the question stops being "does this answer seem competent" and becomes "is this path in the product". The quote forces the judge to look at the thing the criterion is about. Instructions not to be swayed by style ask the judge to suppress an influence it cannot observe in itself; evidence quoting changes what it is looking at.

The table puts a size on that difference, and the sizes are the argument. Telling the judge in so many words that style is not a criterion moved the score from 19 to 20, which is one probe. Requiring the evidence quote moved it from 19 to 23, which is four. Both changes are a few lines of prompt, and the one that worked is not the one that says the right thing. An instruction has to reach the verdict through the judge's own account of why it is grading as it does, and a preference for authoritative prose is precisely the kind of influence that does not appear in such an account — an instruction to ignore it therefore has nothing to bite on. A required span reaches the verdict earlier than that. It settles what the judge must have in hand before it is allowed to answer, and it does that whether or not the judge has any view of its own reasons.

Stripping markdown added nothing once evidence was required, which is worth knowing because it is the fix people reach for first and it is intrusive — it changes what the judge sees relative to what the user saw, and a formatting-dependent criterion would break under it. Single-criterion calls closed the last gap and tripled the cost, which makes them a decision about where the remaining error lives rather than a default.

The tripling is worth following, because it is the same arithmetic as the thirty-five cents and it lands somewhere less obvious. Grading three criteria in one call sends the rubric once and gets three verdicts back. Grading them one per call sends the rubric three times and gets one verdict back each time. The output across the three calls is roughly what the single call produced, since the same three verdicts and three spans are being written either way. What multiplies is the input, and the input is the part that was already most of the bill. That is why the change is fairly described by its call count at all: the part that multiplies and the part that costs are the same part. Had the bill been dominated by what the judge writes, splitting one call into three would have added very little, because the same verdicts and the same spans get written either way.

The limits of a constructed pair#

A probe set is a test for a bias you thought of. It says nothing about the ones you did not, and the space of things a judge can be sensitive to is large: position within the prompt, the presence of a code block, the language of the answer, whether the question was phrased politely. Passing 24 constructed probes establishes that four specific failure modes are not present at the magnitude you constructed, and that is a smaller claim than "the judge is unbiased".

The sentence a reader is likely to write after a clean run is that the judge scored 24 of 24 and is therefore unbiased, and it is worth saying why that is tempting rather than careless. A perfect score on a test normally does license a general claim, and it does so for a good reason: an ordinary test is a sample, its items were drawn from the population you care about, and doing well on a sample is evidence about the rest of what it was drawn from. A probe set is not a sample of anything. Its items were written one at a time by somebody working from a list of four suspicions, and a written item has no population standing behind it. What the score supports is the narrow statement that these four properties did not move this judge's verdicts at the sizes constructed. It supports that narrowly and very strongly, and the strength is what makes it feel as though it ought to reach further.

A probe set is also not a measurement of accuracy. The probes are synthetic pairs with verdicts you assigned, so the judge is being tested against your intent rather than against the distribution of real outputs. A judge can score 24 out of 24 on probes and still be systematically wrong on production traffic, because production contains ambiguous cases that no probe pair represents — the answer that names a setting which existed until last month, the answer that is correct for a different product tier than the user has. Agreement on real, sampled, human-labelled data is the measurement that speaks to that, and it is a different exercise entirely.

The probe set does not transfer, either. It is built from your rubric and your product's failure modes, so it dies the moment the rubric changes and it tells you nothing about a judge for a different task. That is an argument for keeping it small and regenerating it, rather than accumulating a suite that quietly outlives the criteria it was written for.

A probe set is pennies to run, an afternoon to build, and worthless to anybody but you. What it buys is the difference between a judge you have characterised and a judge that has produced plausible verdicts so far. Twelve constructed pairs told this one that it preferred authoritative prose and which single change removed most of that preference — neither of which would have surfaced from grading real items, because on real items the bias and the rubric usually agree.

What the judge can now do is give a single output a defensible score against three conditions. What it cannot do is tell you whether prompt B is better than prompt A, because a score of "two of three criteria pass" for each of them leaves you comparing two numbers that were produced independently and may not be on a stable scale. Deciding which of two systems is better turns out to be a different question from deciding what one system is worth, and the judging protocol that answers it looks different from this one.