Versioning the judge prompt

A metric that lives in a SQL expression gets reviewed as a metric. Somebody notices that the WHERE clause changed, asks what population it now covers, and the pull request acquires a second reviewer. A metric that lives in a judge prompt gets reviewed as prose, because it is prose — a paragraph of English in a markdown file, in a directory that also holds notes and examples, changed by someone improving the wording. Nothing about the artifact signals that it is the definition of the number on the quarterly chart.

Three mechanisms close that gap. A fingerprint — a hash over the prompt exactly as assembled and sent, not over the source file — identifies the instrument that produced every score. A re-score of a frozen anchor set under both the old and the new version turns an edit into a measured delta. And a gate on that delta forces the delta to be acknowledged by a person before it merges. The three together do not stop a rubric from changing, which would be the wrong goal; they make every change deliberate, evidenced and marked in the series.

Three edits in one sentence, each moving a different population#

Here is a rubric edit of the kind that arrives titled "clarify wording".

 ### 3. Policy grounding
-The reply must cite the specific policy section it relies on. A reply that
-refers to "our policy" without naming the section fails this criterion.
+The reply should reference the relevant policy where the customer's question
+depends on it. A reply that grounds its answer in policy satisfies this
+criterion.

Read as a specification change it separates into three independent relaxations. must became should, which converts a hard requirement into a preference the judge can trade off against other criteria. "Cite the specific policy section" became "reference the relevant policy", which drops the specificity requirement and admits replies that gesture at policy without naming it — and the original text had a second sentence spelling out exactly that failure mode, which the edit deleted. And "where the customer's question depends on it" introduces an exemption class that did not exist before: tickets that do not turn on policy now pass this criterion vacuously rather than being scored against it.

Each of those three changes a different thing about how the criterion is applied, which is why they can be counted separately. The must-to-should change alters the strength of the rule, so replies that fail it are no longer automatically failed overall. The specificity change alters the threshold, so a population of replies that used to sit below the bar now sits above it. And the exemption change alters the population the criterion is applied to at all, which is the one that is easy to miss, because a reply that never comes under a criterion and a reply that satisfies it are recorded identically as a pass. A criterion with an exemption clause reports a higher pass rate without a single reply having improved, and no amount of reading the pass rate reveals which of the two it is.

Re-scoring a 240-output anchor set under both judge versions puts numbers on each of those. Version 7 passed 197 of the 240, or 82%; version 8 passed 211, or 88%. The difference between those two counts is exactly 14 outputs, and 14 out of 240 is a shade under 6 points, which is the movement. Fourteen outputs flipped from fail to pass and none flipped the other way, which is what a pure relaxation looks like.

That last observation deserves the emphasis rather than the pass rates, and the reason is that a net movement of 14 and a gross movement of 14 are only equal under one arrangement. Two judge versions that disagree in both directions — some outputs newly passing, some newly failing — would produce a net of 14 out of many more moved items, and the edit would then be doing two things at once, tightening one part of the standard while loosening another. Fourteen and none is a one-directional change, which is the signature of a criterion that got easier and nothing else. It also means the fourteen are worth reading individually, because a small one-directional set is a set with a common cause.

Reading those fourteen, nine were replies that referred to policy in general terms and had previously failed on specificity, and five were on tickets where policy is not at issue at all and which now pass the criterion by exemption. Nine plus five is the whole fourteen, so the two relaxations that could be traced to particular sentences in the diff account for every output that moved, and the must-to-should change turns out to have moved nothing on this anchor set. Six points of movement are accounted for by the edit; there is nothing left over for the product.

That analysis takes twenty minutes once an anchor set exists, and it is the analysis the pull request should carry. A reviewer looking at the diff alone has no way to know whether it moves six points or none — the text reads like a clarification, and the difference between a clarification and a redefinition is empirical.

Stamp the version onto every score you write#

The prerequisite for any of this is that scores carry the identity of the instrument that produced them. Store it with the score, not in a separate changelog that nobody joins against:

{
  "output_id": "out_01J9K2",
  "verdict": "pass",
  "criteria": {"1": "pass", "2": "pass", "3": "pass", "4": "marginal"},
  "judge": {
    "prompt_sha256": "9f2c1ab7d4e08c5f3b6a2d91c7e40f88ab35d2c619fe7a04b8d3c5e21f907a6bd",
    "rubric_version": 8,
    "model": "claude-opus-5",
    "harness_version": "2.4.1",
    "presentation": "both_orders_averaged"
  },
  "scored_at": "2026-06-14T09:12:04Z"
}

The hash is over the assembled prompt rather than the source file, which is the detail teams get wrong. Judge prompts are usually templates: a system block, a rubric file, some few-shot examples pulled from a directory, and an output schema, all composed at runtime. Hashing the rubric file misses a changed schema, a reordered example set, or a system block edited in a different repository, and each of those changes the metric as surely as the diff above.

The rule that decides what belongs in the hash is short: the fingerprint has to identify everything the model reads, because everything the model reads can move a verdict. A source file is one contributor to that, and hashing it answers the question "did this file change" when the question being asked is "would this judge produce the same verdicts". Those two come apart the first time somebody edits a shared system block, and they come apart silently, because the rubric file's hash is unchanged and the fingerprint therefore claims the instrument is the same one.

import hashlib


def judge_fingerprint(system: str, rubric: str, examples: list[str], schema: dict) -> str:
    """Hash exactly what the model will see, in the order it will see it."""
    parts = [system, rubric, *examples, json.dumps(schema, sort_keys=True)]
    digest = hashlib.sha256()
    for part in parts:
        digest.update(part.encode("utf-8"))
        digest.update(b"\x00")   # separator, so concatenation is unambiguous
    return digest.hexdigest()

The zero byte written between the parts is not decoration, and leaving it out reintroduces exactly the failure the hash exists to catch. A hash is computed over a stream of bytes, so feeding it a system block ending in ...check the and a rubric beginning citation produces the same stream as a system block ending in ...check and a rubric beginning the citation. Two genuinely different prompts would then carry one fingerprint, and a query grouping scores by instrument would merge them. The separator is a byte that cannot appear inside any of the parts, so the boundaries between them survive into the stream and two different splits can never collide.

Keep rubric_version alongside the hash even though the hash is the real identifier. The hash is what makes two runs provably comparable; the integer is what a human can say out loud in a review, and a changelog mapping integers to hashes and to the pull request that introduced each one is the artifact that answers "what was the metric in March" in ten seconds rather than an afternoon.

With that field present, every query over historical scores can filter or group by instrument, and a five-month series whose points were produced by four different judges becomes a GROUP BY judge.prompt_sha256 that shows you the version boundaries directly.

The judge is larger than its prompt#

Fingerprinting the assembled prompt catches everything the model reads. A surprising share of metric changes happen outside it, in code nobody thinks of as part of the rubric, which is why harness_version sits in that record next to the hash.

The clearest case is a one-line change raising the judge's max_tokens from 1,024 to 8,192, made because someone noticed truncated reasoning in the logs. It was reviewed as a bug fix, and it was one. It also moved a release gate by two points, because about 4% of judge calls had been hitting the cap mid-verdict, the parser could not read a truncated response, and the fallback path in the harness treated unparseable responses as passes. Fixing the truncation removed forty-odd free passes per thousand outputs. Nothing in the rubric changed; the metric did.

The chain from a token limit to a gate is worth walking link by link, because each link is ordinary and the composite is not. Raising the cap means fewer responses are cut off; fewer cut-off responses means fewer parse failures; fewer parse failures means fewer items taking the fallback path; and the fallback path was recording a pass. So the outputs that had been receiving free passes now receive real verdicts. About 4% of calls were on that path, which is forty in every thousand, and the gate moved by two points rather than four because roughly half of those forty would have passed on the merits anyway. Every step is a consequence of the one before it, and not one of them is visible in a diff that changes a single integer.

What generalises out of that example is a test to apply to any harness setting. Ask what the harness does when the judge fails to answer, and whether that behaviour is the same for every item. A fallback that passes sets a floor under every rate the harness reports; a fallback that fails sets a ceiling; dropping the item changes the denominator instead, which is worse, because the reported rate then describes a population that quietly excludes the cases the judge found hardest.

The list of things in that category is short enough to audit deliberately. What the harness does with an unparseable verdict — fail open and count it as a pass, fail closed and count it as a fail, retry, or drop the item from the denominator — sets a floor or a ceiling on every rate you report, and dropping items silently changes the population rather than the score. Retry policy matters for the same reason: retrying until the judge returns something parseable selects for cases the judge finds easy to articulate. Any per-request setting that shapes how much the judge deliberates belongs in the fingerprint too, as does the output schema, since a schema that asks for per-criterion verdicts elicits different overall verdicts than one asking only for a label. Concurrency and batching are usually safe, with one exception: if your harness runs each comparison in both presentation orders and a change makes it silently stop running the second pass, every margin you publish afterwards is a single-order margin again.

Put those under the same continuous-integration trigger as the rubric file. The cost of over-triggering is a six-minute job on a pull request that turns out to move nothing; the cost of under-triggering is the two points above, discovered a quarter later.

Make the delta part of the review#

The discipline is that a change to any input of that hash triggers an anchor re-score, and the result is posted on the pull request before it can merge.

name: judge-change
on:
  pull_request:
    paths:
      - "evals/judge/**"
      - "evals/schema/verdict.json"

jobs:
  anchor-rescore:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - name: Score the anchor set under base and head
        run: |
          python -m evals.rescore --ref "${{ github.event.pull_request.base.sha }}" \
                                  --anchor anchor_2026_01 --out base.json
          python -m evals.rescore --ref HEAD \
                                  --anchor anchor_2026_01 --out head.json
          python -m evals.diff base.json head.json --comment
      - name: Require acknowledgement for a material shift
        run: python -m evals.gate --max-unacknowledged-delta 1.0

The comment that job posts should carry the pass rate under each version, the paired flip counts in both directions, and the ID of every output whose verdict changed, because the reviewer's real job is to open three or four of those and decide whether the new verdict is the one the team wants. A gate that blocks on a delta above one point unless the pull request carries an explicit label turns a six-point move from something discovered in production into something a human deliberately signed off on.

Not every change to the file needs the full treatment, and forcing it on trivial edits is how a review discipline gets routed around:

One more thing belongs in the repository next to the workflow file, and it is not technical. The rubric encodes a standard the business holds, so the person who owns that standard — a support lead, a product manager, whoever is accountable for what "a good reply" means — should be a required reviewer on it, via CODEOWNERS or the equivalent. Engineering owns the harness, the fingerprint and the re-score job; it does not own the definition of quality, and a rubric change approved only by engineers is a quality standard set by whoever happened to be writing prompts that week.

What the discipline costs, and where a re-score misleads#

The recurring cost is the re-score. A 240-output anchor scored under two versions is 480 judge calls — 240 outputs times two versions — which at roughly 1,600 input and 400 output tokens each comes to about nine dollars and finishes in six minutes at twenty concurrent requests. Forty judge-touching pull requests a year at nine dollars each, with reruns on some of them, lands under four hundred dollars — a rounding error against the salary of the person who would otherwise spend a week reconstructing what happened.

The costs that actually bite are elsewhere. Somebody has to read flipped outputs on every judge pull request, which is fifteen to thirty minutes of attention that a four-minute documentation review did not require. The anchor set ages, and keeping a chain of overlapping frozen sets — a new one each quarter, all of them re-scored on every judge change — is a small standing obligation that nobody owns unless you assign it.

The failure mode with the sharpest edge is a re-score on a stale anchor, because it reports a number rather than an error. If the anchor contains no outputs that exercise the criterion being edited, the job reports a delta of zero and the reviewer concludes the edit was inert — which is the same output the job produces for an edit that genuinely changes nothing.

The reason a zero cannot distinguish those two is worth being precise about, since a green check is exactly what a reviewer will act on. A delta is measured over the outputs in the anchor set, so a criterion can only move it by changing a verdict on one of them. Edit the appeal-window criterion when no archived output in the set mentions an appeal window, and there is no verdict available to change; the job runs correctly, compares two identical result sets, and reports nothing. The measurement is sound and the population is empty, and nothing in the output says which of the two situations produced the zero. Requiring a new judge criterion to arrive with anchor cases that exercise it is the countermeasure, and it is the part of this that needs judgement rather than automation — a machine can check that the anchor was re-scored and cannot check that the anchor contains anything the edit could touch.

Stamping versions does not make old scores comparable. It makes their incomparability visible, which is the most that can be recovered after the fact — a series annotated with four version boundaries is honest, and it is still four short series rather than one long one. Recovering a genuinely long baseline requires re-scoring archived outputs under one fixed judge, and that requires the archive. Versioning also does not stop drift, and it should not try: rubrics legitimately change as products and standards change, and a judge frozen for two years is measuring a standard nobody holds any more.

Most of all, none of this tells you which version is correct. The edit at the top of this chapter relaxed a specificity requirement, and whether that was the repair of an over-strict criterion or the quiet erosion of a standard the business depends on is a question the anchor re-score frames precisely and cannot answer. What the discipline delivers is that the question gets asked in a pull request, by the person who owns the standard, with fourteen flipped outputs in front of her — rather than five months later, by someone reading a chart.