Context window

Read these first

• Tokenization

What you will know

• Record the advertised context limit and verify the effective limit with a fixed prompt and output probe

  Provider documentation may omit wrapper tokens, tool schemas, or hidden reservations that consume capacity before your application content does.

• Measure answer quality as prompt length increases, using a fixed task and an explicit scoring method

  Retrieval and attention failures often appear while the request still fits, especially when relevant material is buried among large amounts of context.

• Calculate how much room is left for output after the prompt is assembled

  Window minus prompt minus reserve. If you have not reserved output space, long prompts silently truncate answers.

• Predict what happens when the window is exceeded, for your provider

  Some providers error, some silently drop the oldest content. Knowing which changes how you defend against it.

• Instrument token usage, reserved output, and overflow outcomes for representative production requests

  Without per-request measurements, prompt growth and provider-side changes can turn into unexplained latency, empty answers, or sudden failures.

Chapters

1. 1

   The assembled request

   A call to a model carries far more than the text you wrote: a chat template applied by the serving layer, a system prompt re-sent every time, all the prior turns your client resends because the provider keeps none, the complete JSON schema of every tool you registered, and whatever retrieval added. All of it becomes tokens in one sequence measured against the context window, which is why registering a single tool whose schema embeds 3,000 product SKUs changes the behaviour of a chat feature nobody edited.

   Read chapter
2. 2

   Probing the enforced context limit

   A context limit is a property of a deployment rather than of a set of weights, so the number on the model card and the number your endpoint enforces routinely differ. Send requests and find the boundary: one probe with a one-token output request and one with a large one, because some endpoints validate input alone and some validate input plus reserved output. Record the advertised figure, the effective ceiling and the date, and treat the output cap as a separate limit with its own error.

   Read chapter
3. 3

   The token budget

   Write the subtraction out with every line visible: a 128k effective window, a 4k system prompt, 6k of tool schemas and a guaranteed 2k answer leaves a little under 116k, and less once the template, the user's message and the conversation history land. The answer is the residual claimant and the only line item that cannot object, so max_tokens does not reserve anything — your assembler does. Compute the remaining room per request and decide in code what gives way when the content will not fit.

   Read chapter
4. 4

   Counting the assembled request

   The number a budget subtracts from is the token count of the fully assembled request — rendered template, role markers, system prompt, resent history and every tool schema — not of the prose you authored. Render the request exactly as it will go on the wire and tokenise that, or hand the identical request body to the provider's token-counting endpoint. Do it at assembly time on every request rather than once as a calibration, because every input to that constant belongs to somebody else.

   Read chapter
5. 5

   Context overflow and silent truncation

   A request that exceeds the window is either rejected with an error naming the count, or quietly shortened by whichever layer of the stack was written to be convenient. The first is a discrete event you can catch, count and act on; the second returns a fluent, confident answer computed over content you believe you sent, and nothing in the response distinguishes it. Establish which behaviour each backend has, normalise to the loud one by checking sizes yourself, and compare the provider's reported input tokens against your own count.

   Read chapter
6. 6

   History trimming policies

   Dropping the oldest messages when the window fills is a cache eviction policy applied to something with no backing store: the model cannot fetch on a miss, and the opening turns are where users state their constraints. The room for history is also the most volatile line in the budget, so a tool whose schema embeds 3,000 SKUs makes conversations start forgetting ten turns sooner with no code change. Pinning, rolling summaries, extracted state and retrieval over history each keep something different, at a different price.

   Read chapter
7. 7

   Effective context length

   A prompt can sit comfortably under the limit and still produce a worse answer than a short prompt containing the same relevant passage, because accuracy declines as material is buried among near-miss distractors. The published benchmarks — NoLiMa, Context Rot, and the effective-length work — agree on the shape and not on the numbers, so the length at which your task stops being answered acceptably has to be measured on your own corpus. Fix the task, fix the scoring, pad with your own retriever's second-best passages, and read the working limit off the curve.

   Read chapter
8. 8

   Instrumenting token usage

   Prompts grow by accretion — a tool here, a few retrieved chunks there, a provider changing its own serialisation — and none of it is visible unless somebody is recording it. Log per request the effective limit, the predicted and actual input tokens, the fixed overhead, the reserve, the output produced and the provider's stated reason for stopping, then alert on headroom rather than on failures. Every field is an integer or a short string, so the record can be kept at full volume and queried by anyone.

   Read chapter