Setting up a context ablation

An ablation is the operation of taking exactly one component out of a system and measuring the system again, so that the difference between the two measurements estimates what the component was doing; the word is borrowed from neuroscience, where a structure is removed and whatever stops working tells you what it was for. Context engineering needs the operation because a prompt gives no account of itself. The window you send is assembled from separable blocks — a system-instruction block, a few-shot block of worked examples, retrieved passages, a conversation tail, a region of tool results — each of which can be taken out with the rest left standing. Reading that assembly tells you what every block says, and it cannot tell you which of them the model is using, since a fluent answer looks the same whether the style guide was followed or ignored.

The measuring is done against a frozen evaluation set — "eval" for short — a fixed list of test cases, each with an input and a standard for a good answer, run through the whole system and scored into one number per run, and edited by nobody while a comparison is in flight. Each configuration it is run against is an arm, so the prompt with a style-guide block and the prompt without it are the two arms of one ablation. The apparatus turns up on ordinary weeks: somebody opens a pull request proposing to delete a block and bank the tokens, a nightly job posts a score four points below yesterday's, two engineers disagree about whether the few-shot examples belong above or below the passages, or a director asks what fraction of the 14,000 tokens sent on every request does any work. Each of those gets settled by an ablation or by whoever argues hardest.

What "everything else" actually contains#

Why the requirement is identity rather than similarity follows from what the two numbers are. Each arm produces a score, and that score is a function of every input the run consumed: the model weights, the decoding settings, the cases, the passages retrieval returned, the rubric the judge applied. Subtract one score from the other and what remains is the combined effect of every input that differed between them, with no term in it labelled by which input produced it. If exactly one input differed, the subtraction has one term, and the difference is an estimate of what that input was doing. If two differed, the difference is their sum — and a sum of two unknowns has no unique decomposition, since the same total is produced by a large positive plus a small negative as easily as by two small positives. That is the entire argument for the one-variable rule. It is a precondition for reading the result at all, not a preference for tidy experiments.

The instinct is to think of the pinned set as "the model and the prompt". In practice the surface is wider, and most of it drifts silently rather than loudly. A model alias is a pointer that the provider is entitled to move. A retrieval index is a live object that ingestion jobs write to. An LLM judge — a language model prompted to score another model's answer against a written rubric — is itself a model call with its own prompt, its own decoding settings and its own drift. The date is an input if your system prompt injects today's date, which most agents do, and a case whose correct answer depends on "within the last 30 days" will change its own ground truth overnight.

Retrieval deserves its own note, because it is the input most likely to be treated as part of the system under test when it is actually part of the environment. If arm A and arm B both call a live vector index, then any ingestion that lands between them changes the passages both arms see, and the ablation is measuring a block removal plus a corpus change. Two fixes work. You can snapshot the index and point eval runs at the snapshot, which is clean but requires your index to support that. Or you can run retrieval once per case, write the returned chunk IDs and text into a cache keyed by case ID, and have both arms replay from the cache — which has the additional benefit of cutting the run's wall-clock time, since retrieval is often a third of it. Replay does mean the ablation can no longer detect anything that changes retrieval itself, which for a prompt-block ablation is exactly right and for a change to the chunker or the reranker — the components deciding how documents are split into passages and which retrieved candidates survive a second scoring pass — would be wrong.

The evaluation seed is worth separating from the model's sampling, because they are different objects that get conflated under the same word. The evaluation seed governs everything your harness does randomly: which cases are sampled if you subsample, what order they run in, which of three judge rubric paraphrases is used if you rotate them, how ties are broken. That seed is yours, it lives in your code, and it is genuinely reproducible. The model's sampling is the provider's, and it is not.

The arms are a written artifact, not a shell history#

An arm is a complete specification of one configuration under test. Writing arms down as data rather than as a sequence of edits to a prompt file is what makes a run reproducible in November that was executed in August, and it is what lets the harness check the one-variable condition mechanically instead of trusting the person who ran it.

# evals/arms/style-guide-ablation.yaml
set: support-eval-v3          # 500 cases
set_sha256: 9f3c1ab7d2e04c85
base: &base
  model: claude-sonnet-4-5-20250929
  temperature: 0.0
  top_p: 1.0
  max_tokens: 700             # the answer budget
  judge_model: claude-sonnet-4-5-20250929
  judge_rubric: rubrics/support-helpfulness-v4.md
  frozen_clock: "2025-08-04T09:00:00Z"
  retrieval_snapshot: idx-2025-08-01
  retrieval_k: 5
  eval_seed: 20250804
  blocks: [system, style_guide, few_shot, passages, conversation]

arms:
  A:
    <<: *base
    label: with-style-guide
  B:
    <<: *base
    label: without-style-guide
    blocks: [system, few_shot, passages, conversation]

The YAML anchor is doing real work: arm B inherits every field from base and overrides one, so the diff between arms is visible in six lines rather than reconstructed by eye from two long configs. That property is worth enforcing rather than hoping for, and it costs about fifteen lines in the runner.

IGNORED = {"label"}

def assert_single_variable(a: dict, b: dict) -> str:
    """Return the name of the one field that differs, or raise."""
    keys = (set(a) | set(b)) - IGNORED
    differing = [k for k in keys if a.get(k) != b.get(k)]
    if len(differing) != 1:
        raise ValueError(
            f"arms differ in {len(differing)} fields: {sorted(differing)}; "
            "an ablation with more than one variable is not attributable"
        )
    return differing[0]

Three decisions inside that function are worth reading rather than skimming. It takes the union of the two arms' field names instead of iterating over one of them, because a field present in A and absent from B is a difference, and iterating over A alone would never look at a field only B has. It drops label from consideration, because the label is the human name for the arm and differs by construction; leaving it in the comparison would make every correctly specified ablation fail. And it rejects a count of zero exactly as firmly as it rejects a count of two. Zero differing fields means the two arms are the same configuration wearing two names, which is a useful run — it is the A/A test the next chapter is built on — but it is not an ablation, and a runner that let it through would report a difference of approximately nothing and invite somebody to conclude that the removed block does nothing.

There is a second reason to keep arms as files in the repository rather than as flags on a command line. The question an ablation answers has a shelf life measured in model releases, and the person who needs to re-open it is usually not the person who ran it. A file in evals/arms/ alongside the set hash, the rubric version and the snapshot ID is a record that survives the laptop it was produced on, and it is the object a decision record will later point at when somebody asks in a design review why the style guide is still in the prompt. Committing the arm file in the same pull request as the prompt change makes the pairing automatic, so the configuration and the claim about it move together through review.

The cost of that discipline is that some runs simply cannot be done: if the model snapshot you measured in August is retired, arm A is not re-runnable, and the honest response is to re-run both arms on the new snapshot rather than to compare August's A against November's B.

Both outputs, every time#

An ablation that reports one number per arm has thrown away the information that makes it actionable. The harness should emit the aggregate and a per-case row, in the same run, joined by a stable case ID, because the aggregate answers "did the system move" and the per-case rows answer "which behaviour moved", and those questions have different answers more often than people expect.

{"case_id":"sup-0142","arm":"A","score":1.0,"input_tokens":13964,"output_tokens":218,"latency_ms":3410,"slice":"refunds/out-of-window"}
{"case_id":"sup-0142","arm":"B","score":0.0,"input_tokens":12461,"output_tokens":364,"latency_ms":3120,"slice":"refunds/out-of-window"}
{"case_id":"sup-0143","arm":"A","score":1.0,"input_tokens":13102,"output_tokens":190,"latency_ms":2980,"slice":"shipping/tracking"}
{"case_id":"sup-0143","arm":"B","score":1.0,"input_tokens":11599,"output_tokens":186,"latency_ms":2905,"slice":"shipping/tracking"}

Two rows per case, one per arm, means the delta is a join rather than a reconstruction, and it means a slice column — the named subset of the eval set a case belongs to, such as refunds/out-of-window — is available without re-running anything. The sup-0142 pair above is the shape that a pure aggregate cannot show you: the case lost a point and its output got 146 tokens longer, which is the signature of a model that started explaining itself instead of following a house format. The requirement here is only that both granularities come out of the same run, because re-running to get the per-case detail after seeing a surprising aggregate reintroduces every drift risk this chapter has just closed.

Where that token figure comes from is worth spelling out, since it is the arithmetic that every per-case read repeats. The two sup-0142 rows carry output_tokens of two hundred and eighteen in arm A and three hundred and sixty-four in arm B, and the second minus the first is the growth. The reason the two facts together say more than either alone is that a wrong answer can be wrong at any length, while a wrong answer that is also substantially longer than the right one is a particular kind of wrong: the model stopped producing the short, formatted response the style guide asked for and started producing an unconstrained one. Set that against sup-0143, which scores one in both arms with output lengths a handful of tokens apart — on that case the removal did not reach anything.

Greedy decoding is not reproducible#

Having pinned everything, it is tempting to conclude that arm A run twice will produce identical output. It will not, and the reasons are structural rather than sloppy. Setting temperature: 0 requests greedy decoding, which selects the highest-probability token at each step, and that is deterministic only if the logits are bit-identical between runs. On a hosted API the logits depend on how your request was batched with other tenants' requests, because floating-point addition is not associative and a different batch shape changes reduction order in the attention and MLP kernels. Mixture-of-experts routing adds a second path: which experts a token is routed to can depend on the other tokens in the batch under capacity-limited routing. Providers also roll hardware and kernel versions underneath a stable model ID. The Anthropic Messages API does not expose a seed parameter at all; OpenAI's seed is documented as best-effort and ships alongside a system_fingerprint you are told to watch precisely because reproducibility is not guaranteed.

That chain is worth walking link by link, because the stumble it prevents is a belief almost everybody starts with. Greedy decoding takes the highest-scoring candidate at each step, which means the output changes only if the ordering of the candidates changes; the scores themselves may wobble freely as long as the winner stays the winner. Each score is the end of a very long chain of sums of products through the model's layers. Floating-point addition is not associative: add three inexact numbers in one grouping and then in another and the two totals can differ in their last bits, because every intermediate sum is rounded back to a fixed number of bits before the next addition happens. The grouping is decided by how the arithmetic was divided across the hardware, that division is decided by the shape of the batch your request landed in, and the batch shape is decided by whichever other tenants sent requests at the same moment. So the scores wobble in their final bits from run to run, and wherever the top two candidates were separated by less than that wobble, the ordering flips and greedy decoding walks down the other branch. The belief to give up is that temperature: 0 means deterministic. It means no sampling, which is a different and weaker guarantee: determinism additionally requires the scores to be bit-identical, and on a shared endpoint you do not control the batch.

The practical size of this is small but not zero, and it is measurable rather than theoretical: on a 500-case support set with a binary LLM judge, running the identical arm twice at temperature 0 typically moves somewhere between six and fourteen cases. That is one to three percent of the set flipping while nothing changed at all, and it means a per-case delta of "nine cases got worse" is indistinguishable from silence until you know your own harness's floor.

An ablation is also not the same instrument as a production experiment, and confusing the two produces a different class of wrong conclusion. A frozen eval set measures a fixed distribution you chose; live traffic measures the distribution you actually get, including the query shapes nobody thought to write cases for. An ablation tells you what a block does on the cases you have, under a judge you wrote, at one point in time. That is genuinely useful and it is not evidence about the tail. Nor does an ablation attribute a mechanism: removing a style-guide block and watching three cases fail tells you the block mattered to those cases, not that the model was reading it — it may have been that the 1,500 tokens were pushing a retrieved passage past a position where the model attends to it, which is a different finding with a different fix.

An ablation is the instrument the rest of this topic is built on, and its accuracy is entirely a bookkeeping property. One variable between arms, every other input pinned to a named version, the arms committed as files next to the set hash they were run against, and both granularities of output emitted from the same execution. Get that right and a two-point difference is a fact about a block; get it wrong and it is a fact about the week.

The open question at the end of a clean ablation is therefore not "is the difference real" but "is the difference larger than what this harness produces when nothing changes". That number is not a constant you can look up, because it depends on your judge, your set, your model and your answer budget, and the only way to have it is to measure it by running one arm against itself.