Green fixtures certify only the properties someone asserted

The change to stage one was four words long. The extraction prompt had been picking up "the customer", "Acme Corp" and "Acme Corporation Ltd" as three separate entities, so someone added an instruction to prefer the most specific mention and collapse duplicates. Stage one's own fixture set — 120 items, field-level F1 against labelled entities — passed at 100%, unchanged. The integration set passed. The change merged on a Wednesday, the nightly end-to-end run came back at 71 against the previous night's 88, and the first message in the channel was that it could not be this change, because the fixtures are green.

Green fixtures were the expected observation, not evidence of innocence. A fixture set certifies the properties somebody thought to write down, and the failure mode that dominates chains is a downstream stage depending on a property that nobody wrote down anywhere. Stage one's metric was F1 over an entity set, which is order-insensitive by construction; the new prompt returned the same entities in a different order; and stage three, four boundaries downstream, had been reading entities[0] as the primary party because for two years the extractor happened to emit entities in the order they appeared in the document. Nothing in the repository ever said so.

What a green suite is evidence of#

The reasoning that sent the channel to look elsewhere is worth writing out, because it is valid in form and wrong in a way that is easy to reproduce. The tests for the edited component passed. Tests failing is how a bad change announces itself. So a change whose tests pass is probably not the bad change. Every step there is reasonable, and the conclusion is still unsound, because the middle step is a claim about all the ways a change can be bad, and a test suite is not a claim about all of them.

What a fixture set actually certifies is the conjunction of the properties its metrics assert, evaluated on the inputs it contains. Stage one's metric was F1 over a set, and a set has no order, so the metric's value is unchanged by any permutation of the output. Rearranging the entities is therefore invisible to it by construction, since comparing two sets is defined without reference to sequence. Carelessness has nothing to do with it; the metric could not have reported a permutation whoever wrote it. The quantity being held constant across the edit was set membership. The dangerous property that varied underneath it was sequence, and the consumer four stages downstream depended on sequence.

The seductive alternative hypothesis — the fixtures pass, so look for another deploy in the same window — sends you to the wrong system for a day. It also gets the epistemics backwards. Passing fixtures raise the probability that the change is fine only in proportion to how much the fixtures assert, and a set that asserts an unordered comparison says nothing whatsoever about ordering.

The dependency itself has an ordinary history, and following it explains why nothing in the repository recorded it. The extractor read the document from beginning to end, so the entities came out in the order they were mentioned. Whoever wrote stage three looked at real output, saw the primary party sitting at the front of the list every time, and wrote entities[0]. At that moment a property of one stage's behaviour became a requirement of another stage's correctness, and the only artefact produced was an index expression that reads as an arbitrary choice.

Walking the localisation#

The first hypothesis, then, is that the edited stage changed some property of its output that its own metric does not measure, and the single check that tests it is a replay: take one input that scored well before and badly now, run each stage on it under both the old and new configuration, and compare the intermediate outputs against the contract each stage promised.

Doing this by hand is a mistake people make once. The chain already writes every intermediate to the ledger, so the replay is a query and a diff rather than four terminal windows:

$ chainctl replay --run 2026-03-18-nightly --record conv-40217 \
    --against 2026-03-17-nightly --show-diff

stage 1  extract      conformance PASS   metric F1=1.00 (was 1.00)
  entities: DIFF (set equal, sequence differs)
    - ["Acme Corporation Ltd", "Dana Whitfield", "Northwind Ltd"]
    + ["Northwind Ltd", "Acme Corporation Ltd", "Dana Whitfield"]
stage 2  normalise    conformance PASS   metric 1.00 (was 1.00)
  account_id: AC-88213 -> AC-31998   DIFF
stage 3  classify     conformance PASS   metric 0.00 (was 1.00)
  category: renewal_question -> billing_dispute
stage 4  draft        conformance PASS   rubric 0.25 (was 0.95)
  reply addressed to the wrong counterparty

Every stage is conformant. Every boundary check from chapter 1 passes, because the payloads are well formed at each joint and always were. The first place a correctness metric moves is stage three, which is where the symptom becomes visible, and the first place anything at all differs is stage one, which is where the cause is. That gap between the two is the whole diagnostic content of the walk: the earliest difference is the suspect, the earliest metric failure is the symptom, and everything between them is a stage that transmitted the problem without noticing.

The rule about the earliest difference is not a heuristic, and it is worth seeing why. A chain is a composition, so a stage's output is determined by its own configuration and its input, and its input is the previous stage's output. If a stage's configuration did not change and its input did not change, its output cannot change. Reading down the replay, then, the first stage whose output differs must be the first place where either a configuration changed or a difference was created, and every difference below it is explained by the input it received. That bounds the search to one stage without any judgement about which stage looks suspicious.

The middle of this walk shows the transmission plainly. Stage two's metric is unchanged at 1.00 and its output differs anyway: it normalised whichever entity it was handed first, so it resolved a different account. A stage doing exactly what it was asked, scoring exactly what it scored yesterday, and passing along a different record is what "transmitted without noticing" means.

The same walk is the correct first move on the other version of this, where a four-stage chain quotes the wrong product tier in customer replies and the drafting prompt reads perfectly — it is told to use the tier it is given. Reading that prompt is where everyone starts, because it is free and it is where the symptom lives, and it is uninformative for exactly the reason the trace above shows: the tier could have been dropped in normalisation, invented in extraction, or overwritten in classification, and every one of those boundaries will have accepted it. Replay one failing record against the contracts before reading any prompt.

Two other explanations produce a similar trace and are worth eliminating in the same pass, since both are common and neither is fixed by editing a prompt. The first is that the input distribution moved rather than the pipeline: a new upstream source started sending conversations in a different format, and the stage that used to handle everything now handles most things. Replaying the old configuration against the new inputs separates the two immediately — if yesterday's code also scores 71 on today's data, nothing you merged is responsible. The second is that a shared dependency changed underneath every stage at once, most often the model version or the judge, which shows up as several stages moving slightly rather than one moving a lot. Recording the model and judge versions in the run metadata makes that a lookup instead of an investigation.

What to look for at each boundary depends on what the symptom is, and the mapping is stable enough to write down:

Read the middle column as a search order rather than as a list of suspects. Each row narrows the chain to the stages that could have produced that symptom, which is what makes the table useful at three in the morning: the symptom is the one thing you always have, and it is enough to rule most of the pipeline out before opening anything.

Who is at fault when nobody asserted it#

Stage three read entities[0]. Stage one stopped emitting entities in document order. Both statements are true and neither stage violated anything it was asked to satisfy, which is why the fault question feels unsatisfying and why teams spend a retro arguing it.

The useful answer is that the fault is located at the boundary, and it is a fault of omission that belongs to whoever built the dependency. Stage three created a requirement on stage one's output when it read an index and interpreted it as meaning "primary party". That requirement is a real constraint on the system whether or not it was recorded, and building it without recording it is the defect. Stage one's edit is the event that revealed it, not the thing that caused it — the same latent dependency would have detonated on a model update, a temperature change, or a re-ordering of the input document.

That said, blame is less interesting than ownership, and ownership runs the other way. The consumer discovers the property; the producer asserts it. The assertion goes into stage one's fixtures, not stage three's, because stage one is the thing that changes and the assertion has to fail at the moment of the edit rather than four stages downstream in a nightly run. A check that lives with the consumer catches the same bug two hours later and does not tell the person who caused it that they caused it.

This is the same argument that produced consumer-driven contract testing between services, where the consumer writes the expectation and the provider's build runs it, and tools like Pact exist to make that flow mechanical. Chains do not have that tooling yet, and the manual version is a comment naming the consumer next to the assertion, which is enough:

class EntitySet(BaseModel):
    entities: list[Entity] = Field(min_length=1)
    # Consumer: stage 3 reads entities[0] as the primary party.
    # Asserted below; do not relax without changing stage 3.

def assert_document_order(out: EntitySet, fx: Fixture) -> float:
    """Entities must appear in order of first mention in the source text."""
    offsets = [fx.input["document"].find(e.surface_form) for e in out.entities]
    return 1.0 if all(a < b for a, b in zip(offsets, offsets[1:])) else 0.0

STAGE_ONE_METRICS = [
    ("entity_f1", field_f1, 0.95),
    ("document_order", assert_document_order, 1.00),   # added 2026-03-19
]

The assertion works by turning the property into a number the existing runner already knows how to handle. It finds where each entity's surface form first appears in the source text, which gives one offset per entity in the order the stage emitted them, and then checks that those offsets increase from left to right by comparing each with the one after it. An increasing sequence of offsets is precisely what "document order" means, expressed in a form a fixture set can score. The threshold beside it is 1.00 rather than 0.95, because unlike a quality metric this is a property that either holds or does not.

Run against the merged prompt, document_order returns 0.0 on 84 of the 120 fixtures, and the four-word edit fails its own stage's set in ninety seconds instead of costing seventeen end-to-end points overnight.

The properties that keep turning out to be load-bearing#

The specific hidden dependency changes; the categories repeat. Across chains the same small set of unstated properties accounts for most of these incidents, and each has a cheap assertion.

None of these is exotic and none of them is caught by a schema check, because in every row the payload remains perfectly well formed. Contracts as chapter 1 defined them constrain shape and stated semantics; these are properties of the distribution of a stage's output, and they need to live in the metric set alongside the correctness measure. The right-hand column is what makes the difference practical: every row's assertion is computed from the fixture's own input and output, with no labelled answer required, which is why this class of check can be added to a stage that has no correctness metric at all.

The coverage row is worth dwelling on because it is the one that hides longest. An extractor handed a 40-page document will often return the same number of entities it returns for a 4-page one, since the instruction says to extract the parties and the model has no notion of how many parties it should have found. Nothing about that output is malformed, every downstream stage accepts it, and the pipeline produces confident, well-structured records that are missing half their content on the largest and usually most valuable inputs. Plotting item count against input length across the fixture set makes it visible in one glance, and turning that into an assertion — the count must not fall as length rises past a threshold — catches the regression that arrives when someone lowers max_tokens to save money.

There is a discipline that makes this converge rather than accumulate forever. Whenever a downstream stage is written to depend on something about upstream output that the contract does not state, the dependency is not allowed to exist until an assertion for it exists in the producer's fixtures. In review that reads as a question — "what are you assuming about the input here?" — and it is the only cheap moment to ask, because the person writing entities[0] knows exactly what they mean by it and will not remember in two years.

The class narrows and does not close#

You can only assert properties somebody has thought of, which means this method is reactive by construction: every hidden dependency costs one incident before it becomes a test. Property-based generation over fixtures, which means deriving test inputs from a rule rather than listing them by hand, helps at the margins — permuting the order of a stage's output and re-running the rest of the chain will find index dependencies without anyone having imagined them — and it is worth doing on the boundaries that carry collections, though the cost of running downstream stages over permuted inputs is real and it finds one class rather than all of them.

Over-assertion has its own failure. An assertion that pins document order forever also forbids the improvement where stage one returns entities ranked by salience, which might be a better extractor. The assertion is not a statement that the property is good; it is a statement that stage three currently depends on it, and the comment naming that consumer is what lets a future engineer relax the assertion deliberately by changing both sides in one commit. Assertions without that provenance calcify a pipeline, and eighteen months later nobody can tell which constraints are requirements and which are fossils.

The chain in this chapter is now maintainable: contracts at the joints, a policy per stage for failures, fixtures per link, and a procedure for localising a regression to the boundary that produced it. That is a substantial amount of machinery around four prompts, and it raises a question the machinery cannot answer. The same task, on the same model, can usually be written as one prompt, and until you have measured what the four stages buy over that one prompt — in score, in tokens, in the ninety-fifth percentile — you are maintaining a pipeline whose existence has never been tested.