Token counting and packing margins

Three token counters given the identical bytes will return three different numbers, and none of them is broken. A local library encoding a serialized payload, a second library shipping the deployed model's own vocabulary, and the provider's counting endpoint are answering slightly different questions, and only one of those questions is the one the create call will ask. The spread is usually a few percent, which sounds negligible and is not, because a packer spends its whole life at a boundary: a four percent disagreement on a 30,000-token prompt is 1,200 tokens, and 1,200 tokens is roughly the size of the candidate the packer is deciding about when the disagreement matters.

The belief that makes this expensive is that a local tokenizer is the count, and it is tempting for better reasons than carelessness. The library implements the same family of algorithm the provider implements, it is deterministic, and it returns an exact integer rather than a range — and exactness reads as authority in a way that anything labelled an estimate never does. What it is exact about is the string you handed it. The provider counts the payload it assembles from your request, which contains that string alongside several other things, encoded with a vocabulary that may not be the one your library shipped. Two exact answers to two different questions look identical on the way past, and neither of them announces which question it answered.

This chapter is about closing that gap and about what to do with the part that will not close, ending in a fill budget where every subtraction has a name and a measurement behind it.

Why a local count drifts from the provider's number#

Four mechanisms produce the drift, and they compose rather than cancel, which is why the total can be larger than any individual cause suggests.

The vocabulary is the first and the largest. A tokenizer library ships a specific vocabulary file at a specific revision, and providers do revise: a model released eighteen months after cl100k_base has different merges, broader coverage of non-Latin scripts, and often a larger vocabulary, so identical text encodes to a different number of pieces. A repository whose comments are written in Korean is that mechanism at its extreme — a generic vocabulary can score Hangul at about 1.5 times what a model with real coverage of it counts. The milder version happens on ordinary English and is easy to miss because it is only one or two percent.

Serialization is the second. When you encode a payload locally you have to decide what string to encode, and the provider decides differently. Whether the JSON is pretty-printed, whether keys appear in the order you wrote them, whether a tool_result array of one text block renders identically to a bare string — these are choices your local code makes and the provider's renderer makes independently. A 74-message transcript can accumulate a few hundred tokens of pure formatting divergence without a single character of your content differing.

Framing is the third: the role markers, turn delimiters and system-block separators enumerated in the previous chapter. What matters here is that a local count reproduces framing only if you have modelled it, and a model calibrated against one endpoint does not transfer to another. The fourth is special-case content — images, PDFs, cached-content markers and citation-enabled documents — which gets counted by rules no text tokenizer implements, so a local count of a payload containing a screenshot is not approximately right, it is missing 1,400 tokens outright.

What stops them offsetting one another is that each acts on a different part of the payload, so none is in a position to undo another. The vocabulary mechanism changes how your content encodes. Serialization changes what string your content is placed inside. Framing adds text your content is not part of at all. Special-case content is counted by rules that have nothing to do with text. A single payload can meet all four at once — a Korean transcript, pretty-printed, carrying a system block and a screenshot — and because the four errors land on disjoint regions they add. Cancellation would need two mechanisms with opposite signs acting on the same region, and three of these four are additions the provider makes that a local count never sees, so their signs mostly agree. Which of the four a given counter is exposed to is decided by what that counter can see:

The last row is not accurate in the way the others are; it is the definition of the number. The endpoint runs the same counting path the create call runs, so agreement is not a property you have to verify. That is the whole reason to use it at the boundary.

"Zero by definition" is a stronger claim than "very accurate", and the difference is the mechanism that makes the whole arrangement work. A token count is a function of exactly two things: the payload as the provider renders it, and the vocabulary the named model uses. The counting endpoint is handed the same request object the create call would be handed, renders it with the same code and encodes it with the same vocabulary, so it is not approximating the create call's count — it is computing it. That is why the last row has no error bar, and why a disagreement between the two would be a defect in the provider rather than a limit of the method. It is also why no better library can replace it: what makes the endpoint authoritative is not the quality of its tokenizer but the fact that it sees the object.

Measure the disagreement before deciding how much to fear it#

Any single figure for the residual is one payload on one model and generalizes to nothing. What generalizes is the method: take a sample of real payloads from production, count each one both ways, and look at the distribution of the residual rather than its mean. A mean error of 1.1 percent is compatible with a p99 of 6 percent — the p99 being the value that one request in every hundred exceeds — and the p99 is what a margin has to survive, since a budget check is a promise made per request rather than on average.

import statistics

def residuals(client, model, payloads, local_count):
    """local_count(payload) -> int, whatever estimator the packer uses."""
    out = []
    for p in payloads:
        truth = client.messages.count_tokens(
            model=model, system=p["system"], tools=p["tools"], messages=p["messages"]
        ).input_tokens
        est = local_count(p)
        out.append((truth - est) / truth)      # positive = estimator ran low
    return out

r = residuals(client, "claude-sonnet-4-5", sample_of_800_requests, packer_estimate)
r.sort()
print(f"mean {statistics.mean(r):+.2%}  p50 {r[len(r)//2]:+.2%} "
      f"p99 {r[int(0.99*len(r))]:+.2%}  max {r[-1]:+.2%}")
# mean +1.34%  p50 +1.02%  p99 +4.10%  max +11.60%

That output is the argument for and against the local tokenizer in one line. Half the time the estimator is within one percent, which is good enough to make packing decisions quickly; one request in a hundred is four percent low, which on a 128,000-token window is 5,200 tokens and a rejection; and the maximum is 11.6 percent, which is a pathological payload — a pasted base64 blob, a generated file — arriving inside a sample of 800. A margin sized at the p99 fails a hundred times a day at scale, and a margin sized at the observed maximum is a guess about a distribution with no bound on the right.

Both halves of that last sentence are arithmetic rather than emphasis. The p99 is by construction the value one request in a hundred exceeds, so a margin placed at it is breached by one percent of traffic — at ten thousand requests a day, a hundred rejections a day, at a rate that does not improve as the system matures, because it is a property of where the margin was put rather than of any defect. The maximum fails in a different way. It is the largest residual in the sample you happened to draw, and a sample of 800 carries no information about the largest residual in a population of millions; drawing another 800 would very likely turn up a bigger one. Sizing a margin at a sample maximum treats an observation as a bound, which is the one thing an observation of an unbounded quantity cannot be.

The 5,200 quoted two paragraphs ago is worth doing rather than reading, since it is the step from a percentage to a rejection. Four percent of 128,000 is a little over five thousand tokens, so a payload the estimator places exactly at the window is over it by five thousand — not marginally, and not in a way the request can recover from. A percentage that sounds like a rounding error becomes a hard failure the moment it is multiplied by a window.

The sample matters as much as the statistic. Drawing 800 payloads uniformly from production gives a distribution dominated by whatever the median request looks like, and the requests that fail are by definition not median. A more useful sample is stratified: a few hundred ordinary requests, plus every request in the last month that was rejected for context length, plus the largest fifty by character count, plus anything from a locale or a content type that arrived recently. That is the sample where the tail lives, and the tail is the only part of the distribution that decides whether a margin holds.

The provider counts once, the estimator counts a thousand times#

Nobody simply calls the endpoint after every candidate addition because a packer is a loop. Filling a 100,000-token prompt from a pool of 400 retrieved chunks means several hundred incremental decisions, and at 90 milliseconds each that is 36 seconds of latency before the model has seen anything, plus several hundred calls against an endpoint with its own rate limit. Four hundred decisions at ninety milliseconds apiece is thirty-six thousand milliseconds, and multiplication is the right operation because the calls are made one at a time: each decision depends on the payload the previous decision produced, so nothing in the loop can be issued in parallel. That dependency is why the latency adds instead of overlapping, and it is why a faster counter would not rescue the arrangement — only a different one would.

The workable arrangement uses each counter for what it is good at: the estimator drives the loop because it is fast and its error is small in the middle of the distribution, and the endpoint decides whether the finished payload is sendable because it is the number that will be applied.

The loop back from the provider count to eviction is what makes the arrangement safe rather than merely fast. If the estimator ran low on this particular payload, the gate — that final provider count standing between the finished payload and the send — catches it and drops material until the real count fits, which costs one or two extra endpoint calls on the small fraction of requests where the estimate was bad and nothing on the rest.

Two details make the gate trustworthy. Count with the model id you are about to send, because counts differ between models and a service that routes between a cheap model and an expensive one has two different numbers for one payload. And count the object you will send, not a copy — build the request dictionary once, pass it to the counter, then pass the same dictionary to the create call, so that no code path can insert a message between the measurement and the send.

What a fill target of ninety percent is claiming#

Writing fill to 0.9 * window looks like caution and is actually an assertion with three parts: that the output reserve is inside the remaining ten percent, that the residual between your count and the provider's is smaller than what is left after that, and that no single item can push past the gap. A packer that reserves nothing for the answer violates the first; a packer that admits a 6,000-line generated file as one candidate violates the third.

A defensible number is derived rather than chosen. Start from the window, subtract the tokens the answer needs, subtract the fixed regions you cannot evict, and turn the residual distribution into an explicit term:

WINDOW        = 128_000
OUTPUT_RESERVE = 8_000     # what the answer needs; chapter 4 is about this number
FIXED          = 4_310     # system block + tool definitions + framing, measured
RESIDUAL_P999  = 0.042     # from the calibration run above
HARD_ITEM_CAP  = 6_000     # no single chunk may exceed this; oversized ones are split

usable = (WINDOW - OUTPUT_RESERVE - FIXED)
budget = int(usable * (1 - RESIDUAL_P999)) - HARD_ITEM_CAP
print(usable, budget)      # 115690 105  ->  115,690 and 104,893

Every term in that calculation is a subtraction except one, and the order the terms are applied in is not free. Taking the 8,000-token output reserve and the four thousand three hundred and ten tokens of fixed regions away from the 128,000-token window leaves 115,690, which is what the code prints as usable. Both of those come off first because neither is negotiable: the answer needs its room, and the system block and tool definitions are on every call whatever the packer decides. The residual is applied to what remains, and as a multiplication rather than a subtraction, because it is a proportional error — an estimator that runs four percent low is low by more tokens on a large payload than on a small one, so the correction has to scale with the thing it is correcting. Taking four point two percent off 115,690 leaves a little under a hundred and eleven thousand. The item cap comes off last and as a flat subtraction, because it is one item's worth of tokens rather than a share of anything, and what survives all four operations is the budget the code prints.

That comes out at 82 percent of the window rather than 90, and every point of the difference has a name and a measurement behind it. When the calibration run is repeated after a model change and the p99.9 residual halves, the budget moves on its own. A hardcoded 90 does not move, and when it turns out to be wrong there is nothing to reason about except whether to make it 85.

The HARD_ITEM_CAP term deserves the emphasis it does not usually get. Percentage margins protect against errors proportional to the payload, and the failure that survives every percentage is a single item whose own count is larger than the gap you left. One 6,000-line generated file, one base64 attachment, one 40-page appendix — the fix is a cap enforced when items enter the pool, with anything larger split or summarized before it is a candidate at all.

The reason a margin cannot be stretched to cover that case is that the two failures are not the same kind of thing. A residual is a counting error: the packer believed a payload was one size and it was another, and a proportional correction answers it because the error scales with what was counted. An oversized item is not a counting error at all. It is a candidate counted perfectly, which happens to be larger than the room left for it, and no adjustment to a term that corrects counts has any bearing on a term that is correct. The two therefore get answered in two different places: the residual at the budget, where a proportion is subtracted, and the item cap at the door where candidates enter the pool, where a size is refused.

The number is honest and still incomplete#

Counting the exact payload removes one class of error and leaves several standing. It says nothing about whether the prompt is any good: a request measured at 104,000 tokens and accepted may still bury its operating instructions under retrieved material the model gives little weight to, and the counter will report a clean fit for a prompt that produces a worse answer than a prompt half its size.

There is a second thing the gate does not do, which is protect a multi-turn agent from its own growth. A payload measured and accepted at the start of a task is not the payload the fourth model call in that task will carry, because each tool result and each assistant turn is appended to the transcript, and an agent that appends a 3,000-token file listing on every step crosses the boundary between two calls that were individually fine. Counting before every call catches that, and it catches it at the worst possible moment — after the tool has run, when the only remaining options are to drop something or to fail. Knowing the growth rate per step is what turns that into a decision you can make early.

Every number in this chapter is about input, and input is one side of the budget. The counter cannot measure what has not been generated yet, so a packer that fills to 104,893 tokens because the arithmetic permits it has made a claim about the 8,000 tokens it reserved that nothing here has checked. What the chapter does settle is the arrangement: a fast estimator for the decisions inside the loop, the provider's count as the single authority on whether the finished object may leave, and a fill target assembled from terms that each move when the thing they measure moves.

If that reserve is too small, the request will succeed, the count will have been correct, and the answer will stop in the middle of a sentence. That is the failure mode where every number is right and the output is unusable, and it is the next chapter's subject.