Context engineering — the budget

Read these first

• Instruction placement and recency
• Token counting and budgeting before the call

What you will know

• Measure and record the token budget for each prompt component: system, tools, retrieval, history, and output reserve

  Use the target model's tokenizer on representative worst-case inputs; character counts and UI estimates are not reliable.

• Show the current allocation for a real feature and defend it

  Tie each slice to observed prompt sizes, answer quality, latency, and the consequence of reclaiming it.

• Decide what gets cut first when the budget is exceeded

  A deterministic policy prevents a large retrieval result or old conversation from displacing instructions or the completion space.

• Measure a request at the context boundary and verify that truncation preserves the configured output reserve

  A request can fit its input window yet leave too little room for a useful completion, causing clipped or incomplete responses.

• Instrument tool-definition tokens and compare them with the rest of the prompt across representative calls

  Verbose schemas, duplicated descriptions, and unused tools consume context on every invocation and can crowd out retrieved evidence.

Chapters

1. 1

   Counting tokens per component

   Characters divided by four is accurate to a few percent on English prose and wrong by 71% on Japanese, 37% on minified JSON and blind to an image, which carries tokens and no characters at all. Count each of the five slices — system, tool definitions, retrieval, history, output reserve — with the target model's own tokenizer, on a worst-case corpus you assemble deliberately rather than on replayed traffic that is missing every request that failed.

   Read chapter
2. 2

   Size guards and context budgets

   Dropping the oldest turns until the joined prompt clears the window has exactly one property: the request goes through. It states no component's entitlement, it runs after retrieval has already committed 11,890 tokens, it rewrites the cached prefix, and it can never protect the completion, which is not part of the string it trims. About twenty lines turn that loop into an allocation whose caps are inputs to fetching.

   Read chapter
3. 3

   The output reserve

   126k of input against a 128k window leaves roughly 2k for the answer whatever max_tokens says, which is how 2% of contract answers stop mid-sentence with a length stop reason and no error anywhere. Raising max_tokens changes nothing and a retry reproduces it at $0.38 a time. Subtract the reserve from the window before assembly, size it from the measured output distribution, and test it at the boundary rather than the median.

   Read chapter
4. 4

   Eviction by component priority

   Truncating the tail of an assembled prompt discards whatever the concatenation order happened to put last, which is how an assistant loses its formatting rules and forgets a tool exists. Separate ordering from eviction: rank components by what the answer cannot be produced without, give each rung a floor, and raise an exception rather than silently trimming. Summarising history lowers how often the ladder fires and does not replace it.

   Read chapter
5. 5

   The tool-definition block

   Twenty-three registered tools came to 6,800 tokens on a 32,000-token window, and with system instructions and the reserve that committed 37 percent of the window before the user's question was considered. The block is byte-identical on every request, so it never becomes a suspect when an answer is bad. Measure it per tool, assert a ceiling in CI, and know that prompt caching lowers its price and not its space.

   Read chapter
6. 6

   Deriving the allocation

   1,500 system, 3,000 tools, 14,000 retrieval, 9,000 history and 4,000 reserve on a 32,000-token model is arbitrary or rigorous depending entirely on what backs each number. Sweep one slice at a time against a frozen eval set, handing the tokens to a competing slice rather than leaving them idle, and take the cap from where the curve flattens. Then record each number with its evidence, its alert and its review date.

   Read chapter
7. 7

   Large context windows

   Passing all forty matching documents because they now fit takes input from 22,000 to 98,000 tokens, which is $6,270 a month against $34,600, a p95 time to first token of 1.9 seconds against 9.2, and grounding of 0.86 against 0.79. A larger window converts a space problem into a recurring cost problem and removes the forcing function that surfaced the retrieval and tool defects underneath. The budget survives; only its ceiling is derived differently.

   Read chapter