Context engineering — selection

Read these first

• Context engineering — the budget

What you will know

• Decide per block whether it is worth its tokens

  A block can be accurate yet lose to a shorter prompt when attention, latency, or cache costs dominate.

• Replace a pre-loaded blob with a just-in-time retrieval, and measure

  Compare answer quality, latency, token spend, and retrieval failure rate on the same evaluation set.

• Construct a case where adding more relevant context lowers answer quality, and measure the crossover

  Late or redundant evidence can dilute salient instructions and leave fewer tokens for the answer.

• Remove something from context and show quality did not drop

  Use a fixed evaluation set and a blinded comparison; unchanged averages may hide regressions in rare cases.

• Recognise a prompt that grew by accretion

  Duplicated instructions, stale examples, and appended exceptions usually signal missing ownership and no context budget.

Chapters

1. 1

   Selecting by marginal contribution

   Raising retrieval from 8 passages to 20 dropped judged accuracy on a frozen 500-question set from 0.78 to 0.71 while a support lead confirmed all 800 retrieved passages were genuinely on-topic. Twenty passages carried six distinct facts in 8,235 tokens; five passages carry all six in 2,090. The test that decides whether a block belongs is not \"is this relevant\" but \"what does this add over the blocks already present\".

   Read chapter
2. 2

   Finding the retrieval crossover

   Somewhere between k=8 and k=20 the accuracy curve turns, and two points do not locate a maximum. Reordering so top-ranked passages sit at both ends cannot find it either: with rank-ordered retrieval the top eight occupy identical offsets in both arms. Here you design the sweep — every arm on the same frozen set, the realised k logged rather than the requested one, and a second pass at a raised budget that separates dilution from eviction pressure.

   Read chapter
3. 3

   Evaluating a block deletion

   Macro-F1 of 0.91 with a six-example block and 0.91 without it is entirely compatible with a rare class losing most of its recall while a common one gains slightly — and few-shot blocks get added for rare, awkward cases in the first place. Here you run the frozen set with and without on identical model and settings, compare case by case rather than on the mean, size the slice that can actually see the class you care about, and ship the deletion so it can be withdrawn without a deploy.

   Read chapter
4. 4

   Pricing a cached prefix

   A projected 4% saving from moving 6,000 tokens out of a cached system prompt is the strongest argument against the change rather than for it: at a 94% hit rate those tokens bill at $0.51 per million against a $3.00 list rate, so 6,000 of them are worth about 1,010 uncached ones. Worse, injecting a 600-token passage ahead of the system block cut input tokens 59% and raised the bill 89%. Read a token count in cached and uncached currency before proposing the cut.

   Read chapter
5. 5

   Just-in-time retrieval and latency

   Pasting a 30,000-token API reference into every request pays prefill and dilution on the questions that need none of it; a search tool pays a retrieval hop and a second model call on the questions that do. Against a 3-second p95 budget where 40% of questions need no endpoint detail, the tool arm comes in at 3.5 seconds and the pasted arm at 2.8. The shipped answer is neither: a 1,850-token pre-load plus a tool, at 2.9 seconds and the best eval score of the three.

   Read chapter
6. 6

   The retrieval miss mode

   Unchanged mean quality with 80% fewer tokens is not yet a case for shipping. Replacing a pasted 12,000-token manual with a lookup tool introduced a failure the old design could not produce — the search returning the wrong rule or nothing at all — and a flat mean is exactly what 198 cases improving by 0.25 and 15 collapsing by 2.20 looks like. Report per-case deltas, retrieval hit rate and a contradiction count before calling this a win.

   Read chapter
7. 7

   Prompt accretion and ownership

   Three separately worded instructions to return JSON, two examples using a priority field dropped from the schema last quarter, and eleven appended \"if the ticket mentions X\" clauses inside 6,200 tokens: every incident was closed by appending. Those symptoms are legible before any eval runs, and the repair — an owner, a per-component budget, provenance on every block — is what makes the deletions defensible. The clean-up took a day and a half and removed 2,220 tokens.

   Read chapter