Cross-family judges and shared blind spots

Two frontier models from different vendors are not two independent instruments. Both were pretrained on overlapping crawls of the same public web, so their factual beliefs about any topic are drawn largely from the same evidence, weighted by the same popularity. Both were post-trained with broadly similar preference-learning recipes on broadly similar notions of helpfulness. Both may have absorbed the same public benchmarks.

The word "independent" is doing precise work there, and the reason the swap disappoints is that two instruments can be different in every visible respect and still fail together. Independence between two graders means that when one is wrong, knowing so tells you nothing about whether the other is wrong. That property is a fact about where their answers come from. A model's answer to a factual question is shaped by what the text it was trained on said, weighted by how often it said it, so two models trained on largely the same text will agree about a fact for the same reason and will agree about a wrong fact for the same reason. Swapping vendor changes who tuned the model and leaves the corpus that supplied the belief almost exactly where it was. What you have bought is a second reading of the same evidence, which is worth something and is not a second source.

What differs between families is the part downstream of that shared base: the specific tuning data, the house style guidelines, the annotation instructions, the length and formatting conventions the model was rewarded for. Those are exactly the sources of a same-family judge's stylistic preference, and the 94-versus-71 gap in the previous chapter attributed seven of its twenty-three points to them, which is why changing family moves that part and only that part. Nothing about changing family touches the shared base. What does not move is therefore the substance of what follows, enumerated as failure classes with the repair that actually addresses each.

Which failure classes survive a vendor change#

Benchmark leakage is the version of this that is easiest to catch and easiest to miss. When a team seeds a golden set, some fraction of the examples are usually adapted from a public question-answering dataset because writing examples by hand is slow. On one such suite, 140 of the 400 examples came from a public source; both judges pass 96% of the answers on those 140 and 84% on the 260 hand-written ones, and the answers are not better on the public slice — the questions are ones both models have effectively seen, so the generator answers them unusually well and the judges grade them against a remembered reference rather than the provided context. Splitting a pass rate by provenance of the example is a one-line group-by that tells you whether the headline is being propped up by material both models already know.

Putting a size on "propped up" takes one weighted average. The 140 public examples are a little over a third of the 400 and pass at 96%; the 260 hand-written ones are the remaining not-quite two thirds and pass at 84%. Multiply each rate by its share and add, and the headline across the whole suite comes out a little over eighty-eight per cent. The number the team would report if the public slice were removed is the 84%, so more than four points of the headline are being supplied by examples that test memory rather than retrieval. That gap is not a small correction on a gate set at 90%, and it moves in the wrong direction over time, because a suite seeded from public data historically grows by adding more hand-written examples, so the headline sags as the suite gets better.

Two properties of that split are what make it evidence rather than a curiosity. The provenance of an example is fixed when the example is written, so it cannot have been influenced by anything the judge did. And both judges show the same split, which rules out the explanation that one family is unusually lenient on this material — a difference both instruments report is a property of the examples.

Seven rows, one of which the swap addresses. That is not an argument against swapping; it is the reason a swap must be priced against a specific failure class rather than bought as a general improvement in evaluation quality. A team that changes vendor and then treats the shared blind spots as handled has bought one row and expensed seven.

Making the judge stop relying on what it knows#

The fourth row is the expensive one, and its fix has nothing to do with model choice: stop asking the judge a question that requires parametric knowledge. Two mechanisms do that, and they cost very different amounts.

The first is a deterministic check, wherever the ground truth exists in a machine-readable form. A product with an OpenAPI specification and a published retention policy can have claims about parameter names or retention windows checked against a file:

import re, yaml

SPEC = yaml.safe_load(open("openapi.yaml"))
PARAMS = {p["name"] for path in SPEC["paths"].values()
          for op in path.values() for p in op.get("parameters", [])}

def unknown_params(answer: str) -> list[str]:
    """Any backticked identifier that is not a real parameter."""
    cited = set(re.findall(r"`([a-z_]{3,})`", answer))
    return sorted(cited - PARAMS)

That function catches a hallucinated since parameter in under a millisecond, with no model, no judge and no ambiguity, and it will do so identically forever. The reason it is exact where a judge is not is that it is not estimating anything. The set of real parameter names is enumerated from the specification file, membership in a set is decided rather than inferred, and no popularity weighting from any corpus enters at any point. Its limits are equally sharp: it sees only what the regular expression's shape of identifier matches, so a parameter mentioned in prose without backticks is invisible to it, and it says nothing about whether a real parameter was used correctly. It catches nothing about a retention window stated in prose rather than as an identifier, which is where the second mechanism comes in.

The second is to put the authoritative source in the judge's context and make the criterion about the source rather than about the world. The criterion changes from "the answer is factually correct" to "every factual claim in the answer is stated in the reference document below", and the judge is handed the relevant section of your own documentation. This turns a knowledge question into a reading-comprehension question, which is the kind of question judges are considerably better at and where shared pretraining beliefs stop mattering — a judge comparing text against a document in front of it does not need to know your retention policy.

The step that makes that work is worth naming, because it is the same step the deterministic check takes by another route. Asking whether a claim is true requires the judge to supply the truth from what it absorbed in training, which is precisely the faculty both families share and both get wrong together. Asking whether a claim appears in a document supplied in the prompt requires the judge to compare two spans of text that are both in front of it. The thing that decides the verdict has moved out of the weights and into the request, so two models with identical wrong beliefs about idempotency keys will both mark an idempotency claim unsupported when the supplied document does not contain it. The correctness of the answer now depends on the correctness of the document you supplied, which is a dependency you control and can audit.

The cost of the second mechanism is retrieval and tokens. The reference section adds 800 to 2,000 input tokens per judge call, which on a 400-example suite with five criteria is between 1.6 and 4 million extra input tokens per run — 400 examples times five criteria is 2,000 calls, and 2,000 calls carrying between 800 and 2,000 extra tokens each spans that range. It also introduces a retrieval step that can itself fail, and a judge given the wrong section will confidently mark a correct answer as unsupported. That failure is worse than the one it replaces, because it is silent in the same way: the judge has a document, the document does not contain the claim, the verdict is a faithful reading of the material it was given, and nothing in the transcript says the material was wrong.

Pricing the swap instead of arguing about it#

A cross-family model may cost several times per token what the incumbent costs, and using it means a new vendor relationship, a security review and a procurement conversation that at many companies runs to six weeks. That is a real price, and the honest question is what fraction of the suite needs to pay it.

Not all five criteria are equally exposed to style preference. C6, the format check, is mechanical and the two judges agree on it 99% of the time; it should not be a judge call at all, since a regular expression settles it. C3, citation presence, is nearly as mechanical once the criterion requires a verbatim span, because you can check that the span appears in both the answer and the context in code. The criteria where the two judges diverged were C1 and C2, the ones that require reading and interpretation, and those are 40% of the judge calls.

The middle rows are where the arithmetic is legible, and following it once makes the last row's claim credible. Moving two of the five criteria into code takes 2,000 judge calls to 1,200, because the two removed criteria were 40% of the calls. Running those 1,200 on a model that costs three times as much per token costs 1,200 times three against a baseline of 2,000 times one, which is 1.8 times the incumbent bill — more expensive than doing nothing, and cheaper than the straight swap in the row above it, which pays the triple rate on all 2,000. The saving comes from the deterministic checks rather than from the judge, and it is worth noticing that those checks are the only line in the table that makes the harness both cheaper and more reliable at once.

The last row is the one most teams should be arguing about, because it treats the cross-family judge as a calibration instrument rather than as the production judge. The cheap same-family judge runs over the full suite; the cross-family judge runs over a stratified 60-example slice — drawn proportionally across question categories rather than uniformly at random, so no category is missing from it — once a week; and the measured offset is published next to the headline pass rate. A gate at 90% against a judge with a known +7 point offset is a gate at 83%, and stating that openly is more honest than a swap nobody can afford to run continuously.

The offset has to be re-measured on a schedule, because it is not a constant. It moves when the generator changes, when either provider ships a model update, and when the criteria are rewritten. A weekly slice costs about 900 judge calls a month, which is small enough that no procurement conversation is required to start it, and having a month of offset data is what makes the eventual procurement conversation short.

Distilled models and judge panels#

Different vendor is not the same as different family in the sense that matters. A large share of open-weight models have been fine-tuned on outputs generated by one of the frontier families, and an instruction-tuned model whose post-training data was distilled from model X will carry X's stylistic preferences into your judge under a different logo. Before treating a swap as a separation, look at how the candidate was trained; the model card and technical report usually say, and where they do not, the absence is informative.

A panel of three judges is not independence either. Averaging three models that share a pretraining corpus reduces the variance of their individual idiosyncrasies and leaves every belief they hold in common exactly where it was, with a majority vote now expressing it with more confidence. Ensembles are worth running for variance, and they are not a defence against a correlated error, which is a distinction that gets lost whenever a team describes a jury as more objective.

The mechanism behind that distinction is the whole reason averaging ever helps, and it is short. Every judge's verdict on an answer is the right verdict plus some error. Where the errors are unrelated to each other, they point in different directions on any given answer, so adding three of them together and dividing by three leaves a smaller error than any one judge had. That is the entire benefit, and it depends completely on the errors being unrelated. Where all three judges make the same error — because all three learned the same wrong retention window from the same documentation — the three errors point the same way, averaging them returns the error unchanged, and the majority vote reports it as unanimous. The panel has improved the part of the problem it could see and left the part it could not exactly where it was, while producing a number that looks better supported.

A vendor swap, then, is a purchase with one line item on it. It buys the removal of one family's house style from the definition of quality, which is a real and measurable seven points on some suites, and it buys nothing against a wrong fact both families believe, a benchmark both memorised, a verbosity preference both share, or a criterion nobody wrote clearly. The repairs for those are cheaper than the swap and none of them involves a procurement conversation: a regular expression, a specification file, a retrieved reference section in the judge's context, and hand-built golden examples that no crawler has seen.

What remains after all of this is a harness with one criterion per call, reasoning before the verdict, deterministic checks where they apply, and a cross-family calibration slice. Every one of those decisions added judge calls, and nobody has yet multiplied the call count by the token price and the number of times the suite runs in a week.