Property assertions on model output

Deleting a golden set leaves a hole where the assertion used to be, and the question a reviewer asks is the right one: if you are not comparing against the recorded answer, what are you comparing against? The answer is the requirements — the things that have to be true of any answer the feature ships, whichever wording the sampler happens to produce this time. On most features those requirements exist, but not anywhere a machine can read them. They live in a product spec, in a launch thread, and in the heads of the two people who wrote the prompt.

This chapter is about writing them down and turning them into checks. It uses a support assistant that answers customer questions from retrieved policy documents — the handful of pages a search step pulls out of a policy corpus for that particular question and pastes into the prompt — and has to cite them. Three things need building: a check function that returns the names of the requirements an answer broke, an aggregation over the whole set of cases that turns per-case verdicts into a rate, and a gate, the rule a continuous integration job applies to that rate to decide whether the build is allowed through.

Writing requirements a machine can read#

For the support assistant, the list comes out looking like this:

• Every factual claim in the answer carries a citation marker.
• Every cited document id is one of the ids retrieved for that query.
• The answer names a policy document at least once.
• The answer does not exceed 1,200 characters.
• The answer never contains the assistant's fallback boilerplate — "as an AI", "I cannot", "consult a lawyer" — because support has a separate escalation path for anything the assistant should not answer.

None of those mention a specific string, and all four of the distinct summaries produced by the five-draw experiment in the first chapter of this topic would satisfy the analogous list for their own feature. That is the property being bought: an assertion that holds for every acceptable answer and fails for an unacceptable one, rather than one that holds for a single acceptable answer and fails against every other.

The mechanical shape is to call the model, parse the output into something typed, then check the parsed object against the requirements. The parse step is not a formality — it is the first assertion, because an answer the harness cannot parse is an answer the service cannot ship.

import re
from collections.abc import Iterable

CITATION = re.compile(r"\[(\d+)\]")
BANNED = ("as an ai", "i cannot", "consult a lawyer")


def violations(answer: str, retrieved_ids: Iterable[int]) -> list[str]:
    """Names of the requirements this answer breaks. Empty list means pass."""
    failed: list[str] = []
    cited = {int(m) for m in CITATION.findall(answer)}
    retrieved = set(retrieved_ids)

    if not cited:
        failed.append("no_citation")
    if not cited <= retrieved:
        failed.append("cited_id_not_retrieved")
    for sentence in split_sentences(answer):
        if is_factual_claim(sentence) and not CITATION.search(sentence):
            failed.append("uncited_claim")
            break
    if any(phrase in answer.lower() for phrase in BANNED):
        failed.append("banned_phrase")
    if len(answer) > 1200:
        failed.append("too_long")

    return failed

The function returns names rather than a boolean, and that choice pays for itself the first time a build goes red. An exact-match assertion produces one bit; this one produces the list of requirements that broke, so the failure says which thing went wrong before anybody opens a diff.

cited_id_not_retrieved is the check worth dwelling on, because it catches the failure everyone is actually afraid of. The model is given, say, documents 4102, 4188 and 4211 for this query. If the answer cites [4102], fine. If it cites [3901] — a real document id, from a real policy, that was not retrieved for this question — the model did not read that document; it produced a plausible-looking id from its own weights. A set containment check, one line long, catches every instance of that failure mode regardless of phrasing, regardless of which id it invents, and regardless of how the surrounding sentence is worded. No golden file can do that, because a golden file only knows about the ids that happened to appear in one recorded answer.

The reason one line covers every instance is that cited <= retrieved is not a heuristic for the failure; it is the failure, restated. Fabricating a source means putting an id in the answer that was not among the ids handed to the model for this query. "In the answer" is the set on the left, "handed to the model" is the set on the right, and "not among" is exactly the question subset containment asks. So the check cannot be evaded by writing the citation differently, because the id is extracted before the comparison and the comparison never looks at prose. It cannot be evaded by inventing a different id, because every id outside the retrieved set fails equally. And it does not need updating when the corpus grows, because the right-hand set is rebuilt per query from what the retriever actually returned on that call.

The intuition to carry away from that is the one this whole chapter turns on. A check written against a property of the answer holds for every wording of it, because wording is not what it reads. A check written against a stored string holds for one wording, because that is all it has.

Two of the requirements in that check function can be moved out of it entirely by constraining the output format at the API. If the response is a JSON object with a schema, "required fields present" and "the routing field is one of four values" stop being assertions and become guarantees:

response = client.messages.parse(
    model="claude-opus-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": prompt}],
    output_config={"format": {"type": "json_schema", "schema": ANSWER_SCHEMA}},
)

That is a genuine reduction in what the tests have to cover, and it is worth doing before writing checks that duplicate a schema. What it cannot constrain is anything about the content of the strings inside the object, which is where the remaining requirements live.

Four families, and what each is blind to#

The checks above are not all the same kind of thing, and grouping them makes the coverage gaps visible:

The first three are free and fast, which is why a suite should exhaust them before reaching for the fourth. Two thousand cases of structural and referential checking costs nothing beyond the model calls that produced the answers; the same two thousand cases judged by a judge model — a second model call whose only job is to read an answer and score it against a rubric, a short written list of criteria in plain English — roughly doubles the bill and adds a second sampled generator whose own variation you now have to reason about.

The blind-spot column is the honest part. cited_id_not_retrieved proves that a cited document was in front of the model. It does not prove the document says what the sentence claims it says, and an answer that cites document 4102 beside a sentence 4102 contradicts passes every check in the function. Blocklists have the standard blocklist weakness: "consult a lawyer" is caught, "you may wish to seek legal advice" is not, and the list only ever grows by incident. These are real limits rather than reasons to skip the checks — a cheap check that catches the fabricated-id class completely is worth more than an expensive one that catches everything at 80%.

From one verdict to a rate#

A per-case verdict is still one draw, and a single call's result is noise. The harness therefore runs the checks over every case and reports what fraction passed:

Note where the parse failure lands. An unparseable response is scored as a failing case, not as a harness error swallowed in a try block, because a service that cannot parse its own model output is broken in exactly the way the suite is meant to detect. Provider errors are the opposite and belong in a separate bucket: a 429 is a fact about your rate limit, not about the prompt, and counting it as a case failure moves the reported rate for reasons that have nothing to do with quality.

The aggregation is unremarkable code and the output is the interesting part:

from collections import Counter

results = [violations(*run_case(case)) for case in load_cases("cases/*.json")]
passed = sum(1 for failed in results if not failed)
by_requirement = Counter(name for failed in results for name in failed)

print(f"{passed}/{len(results)} = {passed / len(results):.3f}")
for name, count in by_requirement.most_common():
    print(f"  {name:<26} {count}")

131/150 = 0.873
  uncited_claim              11
  cited_id_not_retrieved      6
  too_long                    3
  no_citation                 1
  banned_phrase               0

Nineteen cases failed and twenty-one requirement violations were recorded, because two cases broke two requirements each. That breakdown is the artifact worth having. It says the dominant failure is claims without citations, that six answers cited something the retriever never returned, and that the boilerplate blocklist caught nothing at all — which is either good news or evidence that the blocklist is checking for phrasings the model stopped producing months ago.

The gate is then a threshold on the rate rather than a demand that every case pass. Requiring 150/150 puts you straight back in the world of exact matching: with a per-case pass rate of 0.99 on every case, the probability that all 150 pass on a given run is 0.99¹⁵⁰ ≈ 0.22, so a genuinely excellent system fails the gate four builds in five. A threshold accepts that the measured quantity is a rate and asks whether the rate is high enough.

That exponent deserves unpacking, because "0.99 is nearly one, so 0.99 to any power is nearly one" is the intuition almost everybody arrives with and it is wrong. Start with two cases. Each passes with probability 0.99, and if they pass independently — which they do here, since each case is a separate model call with no state shared between them — then both passing is 0.99 multiplied by 0.99, a little over ninety-eight times in a hundred. Ten cases is 0.99 multiplied by itself ten times, which is around nine runs in ten. The pattern is that each additional case multiplies the survival probability by 0.99 again, so what looks like a rounding error compounds: by 150 cases the product has fallen to about 0.22, and the all-pass run has become the exception.

The same fact is easier to feel from the other side, and the second reading is the one to keep. A per-case failure rate of one in a hundred, across 150 cases, produces about one and a half failing cases on an average run. Demanding 150/150 is therefore demanding a run with no failures when the expected number is above one — which is not a quality bar at all, it is a request for a lucky draw. Nothing about the system needs to change for the gate to flip between red and green; only which cases happened to land badly this time.

The way property checks go wrong#

The failure mode worth naming is over-specification, because it is the natural instinct of someone who has just deleted 150 golden files and feels the loss of coverage. The checks creep back toward the string. A requirement that the answer "starts with a direct statement of the policy" becomes a regex for ^The (refund|return) policy; a requirement that it be concise becomes a check for exactly one paragraph; a requirement that it be polite becomes a list of approved opening words. Each addition looks like tightening, and collectively they reconstruct exact matching with more moving parts and worse error messages. The test is whether a rewrite a reviewer would happily ship would fail the check. If it would, the check is pinning phrasing, and it will produce the same false failures on the same schedule as the golden file it replaced.

The opposite failure is a suite so loose it passes garbage. "The policy is [4102]." satisfies every check in the function above — it cites a retrieved id, carries a citation on its only claim, is under 1,200 characters, and contains no banned phrase. It is also useless to a customer. Free checks cannot detect that, which is precisely the gap the judged family exists to fill, and the reason a suite of structural checks is a floor on quality rather than a measure of it.

The precise version of "floor" is worth stating, because a team that has just built these checks will be tempted to read a green run as a verdict on the prompt. Every requirement in the function is a necessary condition: an acceptable answer must satisfy it, so failing one proves the answer is unacceptable. None of them is sufficient: satisfying all five does not prove the answer is acceptable, because the list was written by people and contains what they thought of. Those two statements are not symmetrical, and the asymmetry is permanent. A red run is evidence about the system. A green run is evidence about the list — it says nothing broke among the things you decided to watch, which is exactly as strong as the watching.

That is why the sentence a reader is most likely to write after building this suite is the one to be careful about. "The checks pass, so the prompt is right" does not follow from anything above. What follows is narrower and still worth having: no answer in this run violated a requirement anybody has written down yet, and the requirements that exist were mostly written after somebody was burned by their absence.

Requirements also decay, and the sources they come from say how. A product spec and a customer-facing policy are stable; incidents are not. Every escalated conversation the checks let through is a requirement nobody had written down yet, and adding it is the durable half of an incident response. Publishing the per-requirement breakdown keeps the other direction honest too: a requirement that has recorded zero violations for four months is either genuinely solved or no longer checking anything, and it is worth finding out which.

This is also the answer for a stranded case like the refund-window assertion from the previous chapter, which could not be quarantined without going blind and could not be retried without discarding the number. It stops being a boolean assertion about one ticket and becomes one case in a batch — sampled several times, scored against the requirement that the deadline is legible, reported as a rate a threshold can gate on. What comes out then means something: 0.85 is a statement about behaviour, where pass was a statement about luck.

None of that, though, settles the pressing question about 131/150 = 0.873. That figure was computed over 150 cases, and nothing so far has asked where the 150 came from, who wrote them, or whether 150 is enough to distinguish a system that fails 3% of the time from one that fails 15% of the time. The next chapter does that arithmetic, starting from a much smaller number of cases and a launch that went ahead on the strength of them.