Token counting and budgeting before the call

Read these first

• Token economics
• Conversation state

What you will know

• Count tokens for a prompt before sending it

  The provider's tokenizer and message framing determine the real limit, and wrappers may add tokens that a local character count misses.

• Enforce a budget that reserves room for the output

  A prompt can fit the input limit yet leave too little space for a useful completion, especially when the model's maximum output is capped separately.

• Truncate history without breaking the message structure

  Tool calls, tool results, and assistant turns often form required pairs, so removing one side can cause API rejection or misleading model context.

• Choose what to drop first when the budget is exceeded

  Eviction policy affects answer quality: system constraints and recent task state usually matter more than stale turns, redundant retrieval, or illustrative examples.

• Compare token counts from two tokenizers against the provider's count

  Characters-per-token estimates diverge sharply for code, emoji, non-English text, and long identifiers, making boundary decisions unsafe without measuring the actual tokenizer.

Chapters

1. 1

   Characters per token

   Dividing a character count by four matches the provider within two percent on English prose and collapses on Java stack traces, minified JSON and Korean comments, because camel-case identifiers, dense punctuation and under-represented scripts all fragment into far more tokens for the same characters. The error is a function of the content, so no fixed safety margin removes it: this chapter measures the signed error per content class, shows why a correctly sized margin costs a third of the window, and explains why a better estimator is still an estimator.

   Read chapter
2. 2

   Tool definitions and request framing

   Counting the concatenated content of your messages and reserving the remainder still gets requests rejected by a few hundred tokens, because role framing surrounds every message, the system block is its own segment, and tool definitions carry JSON Schema descriptions that run into the thousands of tokens on their own and are resent on every call. This chapter enumerates everything the provider counts that a string concatenation never saw, breaks a tool definition down into the parts worth editing, and measures the per-message framing exactly.

   Read chapter
3. 3

   Token counting and packing margins

   Two local tokenizer libraries and the provider's own counting endpoint can disagree by four percent on a 30,000-token prompt, which is 1,200 tokens of uncertainty at exactly the boundary a packer is deciding. This chapter explains the four mechanisms that produce the drift, shows how to measure the residual distribution on real payloads rather than trusting a mean, and derives a fill budget with a named term for every subtraction — including a per-item cap, which is the one failure no percentage margin covers.

   Read chapter
4. 4

   The context window and maximum output

   Input and output draw on one pool, so a packer that fills to the input limit and sends produces successful calls whose answers stop mid-sentence — a failure that never appears on an error-rate graph because it is reported in a stop_reason field. Budget backwards from a measured reserve instead, and set max_tokens to that reserve so an overrun fails loudly. Maximum output is a second, separate cap that does not grow when the input is small, which is why a model that can read a sixty-page contract may be unable to rewrite it.

   Read chapter
5. 5

   Decomposing an output-bound job

   When the deliverable is longer than any available model can emit, no reserve computation and no eviction policy closes the gap, because reserving divides a window rather than enlarging one. Decomposition splits the work into groups that each fit an output cap, sends the surrounding context each group needs, and stitches the results. This chapter derives the group size, prices the duplicated shared context, compares four ways of dividing the work, and treats the seams — terminology drift, renumbering, broken cross-references — as the part that becomes your problem.

   Read chapter
6. 6

   Compaction with tool calling

   Evicting oldest-first works on a chat transcript and breaks as soon as tool calling arrives, because tool calls and their results reference each other by id and the API rejects an array where either half is missing. Compaction therefore has to remove whole exchanges rather than individual messages, a validator in the request path turns the remaining 400s into test failures, and the tool-execution path has to append a result even when the tool raised. Raising the compaction threshold only makes the failure rarer and harder to reproduce.

   Read chapter
7. 7

   Eviction order under a token budget

   Ranking removable material by size produces an agent that drops its own operating constraints to make room for a log file. Rank by recoverability first, then staleness, then how directly the material bears on the decision the model is about to make, and only then by size — and replace large tool results with a short durable record rather than deleting them outright. This chapter gives the order, the symptoms of a wrong one, the evidence that should make you reorder it, and what a larger window does and does not fix.

   Read chapter