Context engineering — measurement

Read these first

• Context engineering — eviction and compaction
• Component vs end-to-end evaluation

What you will know

• Ablate a context block and re-run the eval to see if it mattered

  A block can help one slice while harming another, so compare aggregate and per-case results under the same model, prompts, and evaluation seed.

• Show a measured case of context rot

  Track quality against increasing context load while holding the task and answer budget steady; stale or distracting material should produce a visible degradation pattern.

• Report what fraction of supplied context the task actually needed

  Define “needed” with a reproducible removal or masking test, because token count alone cannot distinguish useful evidence from tolerated noise.

• Turn a context-engineering opinion into a measured decision

  Record the competing variants, evaluation threshold, and operational constraint that determine whether a block is retained, reordered, or removed.

• Instrument context quality and cost in the evaluation run

  Capture token counts, retrieval or selection provenance, latency, and per-example scores so regressions can be attributed rather than guessed at.

Chapters

1. 1

   Setting up a context ablation

   Removing one block from a prompt and re-running tells you what that block was doing only if the model, the prompts, the retrieval index, the judge, the answer budget and the injected date are identical across arms. This chapter sets the ablation up as the core instrument of context engineering — one variable at a time, a frozen set with a hash, arms written down as files, aggregate and per-case output from the same run — and is equally honest that pinning everything still leaves a provider's sampling non-reproducible from one run to the next.

   Read chapter
2. 2

   The churn baseline

   Before per-case deltas mean anything you have to know how many cases flip when nothing changes at all. Running one arm twice under identical settings and counting the cases whose score moves gives a floor that only the excess over it can be read as a finding. This chapter measures the floor, splits it into the four sources that produce it, prices the repeats that shrink it, and shows why a response cache appears to drive it to zero without doing so.

   Read chapter
3. 3

   Per-case deltas and cancellation

   0.84 before and 0.84 after is what you see when a block does nothing, and equally what you see when it helps one group of cases and harms another by the same amount. On a 500-case set, 41 improved and 39 worsened with 420 identical is a large effect that happens to sum to zero. This chapter gives the fixed sequence of questions to ask once an aggregate has not moved, the grouping query that separates the two directions, and the weighting that turns a tie into a decision.

   Read chapter
4. 4

   Instrumenting an eval run

   Four points lost overnight with nobody able to name the cause is a harness defect rather than a mystery: an aggregate is a gauge, and the columns that would have separated the hypotheses were discarded milliseconds before the score was computed. This chapter specifies the row an eval run writes per case — per-block token counts, retrieval provenance with gold rank, judge votes, truncation, latency and the configuration that produced them — and shows why cost and quality have to land in the same row to be joinable at all.

   Read chapter
5. 5

   Component evals and prompt changes

   Delete a 2,000-token glossary and recall@5 will not move, because retrieval never saw it — the metric was structurally incapable of moving rather than reassuringly flat. A metric only detects a change that occurs upstream of, or at, the boundary it scores, which puts every prompt-assembly change outside a retrieval metric's field of view and inside groundedness's. This chapter maps the boundaries, works the four cases behind a one-point end-to-end drop, and says what a moved recall number would have meant instead.

   Read chapter
6. 6

   Needle tests and context rot

   Verbatim recall of a planted fact saturates well past the point where multi-hop reasoning over the same input starts failing, so running a needle-in-a-haystack test on your own corpus settles nothing about whether volume is hurting you. This chapter designs the experiment that does settle it — fix the questions, the evidence and the answer budget, vary only the surrounding material, run a needle control alongside single-hop and two-hop families — and lists the five confounds that produce a curve meaning something other than what you will read into it.

   Read chapter
7. 7

   Pre-registering a threshold

   Evidence does not become a decision without a rule for turning one into the other, and that rule has to exist before the numbers do. Write the competing opinions down as named variants with arm files, agree the metric, the minimum effect, the operational constraint and the action on each outcome — including inconclusive — then run. This chapter works an ordering question whose answer turns out to be about prompt caching rather than attention, and prices a 1.5-point quality gain that clears its statistical bar and fails its business one.

   Read chapter
8. 8

   The needed context fraction

   A block is needed if removing it and re-running the frozen eval drops the score below a threshold agreed in advance, and a leave-one-out sweep over every removable unit turns \"what fraction of our 14,000 tokens does any work\" into a number produced in an afternoon of compute. This chapter runs the sweep, decomposes the blocks that matter, shows why marginal contributions do not add up to a plan, and attaches the two conditions — an oracle measurement and a coverage gap — that have to travel with the number or it becomes a slide saying 70% is waste.

   Read chapter