Characters per token

The 128,000 in "a 128,000-token model" is a context window: the fixed number of tokens one request may occupy, shared between everything you send and everything the model writes back. It exists because a model reads a sequence of positions whose length is fixed when the model is built, so the limit is structural, and a request that exceeds it is refused outright with an HTTP 400 rather than quietly trimmed to fit. The provider decides how large your request is by running its own tokenizer, the program that cuts text into the pieces the model reads. Working that count out before you press send, and choosing what to include so the total lands under the window with room left for the reply, is token budgeting, which is what this topic is about. The code that does it is a packer: a loop that adds material to the prompt — retrieved documents, earlier turns of a conversation, diff hunks, the files an agent has read — until the budget is spent, and then stops.

Calling the provider once per candidate is far too slow for a loop of a few hundred iterations, so packers size candidates with a local estimator instead, and len(text) // 4 is the cheapest estimator available. This chapter is about what that divisor is actually measuring and where it stops holding. It puts a signed error against six classes of real content, works out what a margin would have to be to survive the worst of them and what that margin costs on ordinary traffic, and then shows the same failure surviving an upgrade to a genuine tokenizer — because the shape of the problem is a residual between an estimate and a payload, and a better estimator narrows the residual without removing it.

The divisor is a fact about the text you measured it on#

Dividing by four is a compression ratio borrowed from a body of text, and it holds for exactly the kind of material that text contained. A tokenizer's vocabulary compresses what it saw often in training and reassembles everything else out of smaller pieces, so indentation, punctuation clusters, split identifiers and under-represented scripts all fragment.

That sentence is the foundation of everything below it, and it is worth taking one layer further down. A vocabulary is a fixed inventory of text fragments, settled once when the tokenizer is built, and the fragments in it are the ones that recurred often in the body of text it was built from — whole common words, common word endings, common runs of punctuation. Encoding a string means covering it with entries from that inventory, and anything the inventory does not already contain has to be spelled out of the smaller pieces that it does contain. So the number of tokens a string costs is decided by how much of that string the inventory already holds, which makes characters per token a measurement of how closely the text resembles the corpus behind the inventory rather than a property of text as such.

Four is therefore a number about English prose, and it holds while the content keeps resembling English prose. The wrong belief to name here is the one a reader carries away from any first encounter with a token: that counting characters approximates counting tokens. It is tempting for three good reasons — both quantities are lengths of the same string, the ratio between them really is stable inside a single class of content, and a validation run on real data will confirm it convincingly. It is wrong because the ratio is a function of what the content is made of, and what the content is made of is chosen by whoever is typing rather than by whoever wrote the packer.

What matters for budgeting is the shape of the resulting error, so the useful table is not chars-per-token but the signed error the estimator makes against the real count. That error is the estimate minus the real count, divided by the real count, so a positive number means the estimator ran high and a negative one means it ran low. Dividing by the real count rather than by the estimate is what makes the rows comparable, because the real count is the quantity the provider will act on:

Two rows worked in full make the other four readable. The English row divides 12,400 characters by four to get 3,100 against a real count of 2,940, so the estimate is a hundred and sixty tokens high, and a hundred and sixty over 2,940 is a little over five in a hundred — the +5.4 percent. The stack-trace row divides 8,150 by four to get 2,037 against a real count of 3,510, so the estimate is a little under fifteen hundred tokens low, and fifteen hundred over 3,510 is about forty-two in a hundred — the −42.0 percent. Nothing in either calculation is specific to the sample; it is the same two operations against whatever pair of counts a new content class produces.

A positive number in the last column means the estimate was high, which is harmless because it makes the packer stop early. Every other row is an under-count, and an under-count is what produces a rejected request. The 42 percent on the stack-trace row is the one that breaks a support tool: eleven traces of that size are about 38,600 real tokens where the estimator sees 22,400, and 16,000 tokens of invisible material is more than any margin anyone would have written down.

The first two of those figures are multiplications and the third is the subtraction between them. Eleven traces at 3,510 real tokens each is a little over thirty-eight and a half thousand, which is where the 38,600 comes from. Eleven at the estimator's 2,037 is a little over twenty-two thousand four hundred. Taking the second from the first leaves about sixteen thousand tokens of material that exists in the payload and not in the packer's arithmetic. The multiplication is allowed because tokenization is applied to each piece of text on its own — the tokenizer does not compress the second trace against the first, so eleven copies of a thing cost eleven times what one copy costs, and an error made once is an error made eleven times.

The Korean row is the one that looks impossible, since it reports fewer than one character per token, and the reason is worth having because it is the same mechanism as the others taken to its end. A vocabulary assembled mostly from Latin-script text holds few or no whole Hangul syllables. Every tokenizer also keeps a set of byte-level fallback pieces, so that any input at all can be encoded rather than rejected, and text the vocabulary does not cover falls through to those. A Hangul syllable is more than one byte, so encoding it means emitting more than one fallback piece, and a string whose characters each cost several tokens has a chars-per-token figure below one. There is nothing pathological about the text. It is ordinary Korean meeting an inventory that was not built to hold it.

Two different mechanisms produce the code rows, and it is worth keeping them apart because they scale differently. Identifier splitting raises tokens per word: LedgerService is three pieces, applyCredit is two, so a line of Java carries roughly twice the tokens per word that a line of English does. Punctuation density raises tokens per character, because a fragment like .java:412) has almost no reusable substrings in it and lands close to one token per character. A stack-trace line is unusual in being bad at both at once:

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
line = "	at com.acme.billing.internal.LedgerService.applyCredit(LedgerService.java:412)"

ids = enc.encode(line)
print(len(line), "chars ->", len(ids), "tokens")
print([enc.decode([i]) for i in ids])
# 78 chars -> 24 tokens
# ['\t', 'at', ' com', '.', 'ac', 'me', '.', 'billing', '.', 'internal', '.', 'L',
#  'edger', 'Service', '.', 'apply', 'Credit', '(', 'L', 'edger', 'Service', '.',
#  'java', ':', '412', ')']

Repeated thirty times, that structure is the whole trace, and the estimator sees a comfortable 78 characters per line where the provider sees 24 tokens. Spanish is milder and moves the same way: accented characters and the endings that Spanish uses heavily — configuración, actualizaciones, proporcionado — sit further from the vocabulary's common merges than their English counterparts, which costs roughly 15 percent more tokens for the same idea.

A margin cannot absorb an error that moves with the content#

The reflex after a rejection is to widen the margin, and the arithmetic of that reflex is worth doing explicitly, because it is the reason the reflex does not work. If the estimate can be as low as 58 percent of the true count, then to be safe you must treat every estimate as if it were 58 percent of the truth, which means you may only fill 0.58 × window when the estimate is what you are checking. On a 128,000-token window that is 74,000 tokens of usable space, and 54,000 tokens you have paid for and cannot use on the English requests that make up most of your traffic.

The 58 comes straight out of the table and is worth carrying across explicitly, because it is the step where a percentage error becomes a fill target. An error of −42 percent means the estimate is the true count less forty-two hundredths of the true count, which is fifty-eight hundredths of it. Turning that around, the true count can be as large as the estimate divided by fifty-eight hundredths, so an estimate that is safe against the worst row is one whose worst-case true count still fits the window. Multiplying the window by fifty-eight hundredths is what produces a ceiling with that property, and 58 hundredths of 128,000 is a little over seventy-four thousand. The 54,000 is the remainder, and it is unusable rather than merely unused: on an English request the estimator is accurate, so those tokens would have held real material and the margin forbids them anyway.

Those surrendered 54,000 tokens are the cost of a margin sized correctly. The margin sized the way people actually size it — ten percent, because ten percent feels careful — buys nothing on this failure, since the error it needs to absorb is four times larger. Between those two positions there is no good place to stand, because the error is not a constant with noise around it; it is a function of the content, and the content is chosen by whoever is typing.

The tail is the part that finishes the argument. The table above is a set of averages over whole samples, and a request is not an average — it is a specific set of pieces, one of which may be a base64-encoded certificate a user pasted into a support ticket, or a minified bundle, or a Unicode table. A single 40,000-character blob of base64 encodes to roughly 13,000 tokens at about 3.0 characters per token — 40,000 divided by three is a little over thirteen thousand — where len(text) // 4 reports ten thousand, so one pasted attachment overflows a budget by three thousand tokens on its own. One emoji sequence with a skin-tone modifier and a zero-width joiner is four to eight tokens for what a len() call counts as three or four characters, which is the same failure at a size that would never be noticed and in a direction that compounds with every other emoji in the message. No percentage subtracted from the window covers the pathological item, because the pathological item's error is unbounded above.

Unbounded above is the precise claim and it is worth separating from "large". A margin expressed as a share of the window is a fixed number of tokens once the window is fixed. The error a single item can contribute is not fixed by anything, because the item's size is chosen by whoever pasted it and its characters-per-token ratio is chosen by what kind of bytes they pasted. There is therefore no percentage with the property that every item fits under it, and the search for one is a search for a bound on a quantity that has none.

Two repositories, one cause#

A code-review bot makes the same mistake at a higher level of sophistication, which is why it is the version worth studying. Its packer fills the prompt with diff hunks up to 90 percent of the window, counting each hunk with a local tokenizer rather than a divisor. On a large Java service the requests started coming back rejected for context length. On a different repository, one whose comments are written in Korean, the requests went out fine and turned out to be about 60 percent full — a third of the window bought and left empty on every review.

The same estimator overflows the window on one repository and leaves a Korean-commented one about forty percent empty, and those look like opposite bugs. They are one bug. The number the packer trusts is an estimate of the payload rather than a measurement of it, and the residual — the gap between what the estimate says and what the provider will count — is a function of what the content is made of, including its sign.

On the Java repository the residual is positive, meaning the payload is bigger than the estimate. The estimate counts hunk bodies, while the prompt renders each hunk with a file path, a line-number header and a fenced block, and Java package paths are long: src/main/java/com/acme/billing/internal/LedgerServiceTest.java is 60 characters and about 20 tokens. Twenty tokens repeated for each of 400 hunks is 8,000 tokens the count never saw, before the fences and the per-file instruction are added. Nothing here is an error in the tokenizer. It encoded exactly the strings it was handed, and the strings it was handed were not the payload.

On the Korean repository the residual is negative, and for a different reason. The local tokenizer is cl100k_base — the default almost everyone reaches for — while the deployed model uses a newer vocabulary with real Hangul coverage. Korean text that the local library scores at 2,480 tokens is 1,560 tokens to the model. Dividing the first by the second gives about 1.5, so the packer's count runs about half as high again as the truth on the comment-heavy portion of each hunk, and a count that runs high stops the loop early. It halts at what it believes is 90 percent of the window; ninety divided by one and a half is sixty, so the payload that actually leaves is about 60 percent.

# The packer's view and the provider's view of the same 400 hunks.
local_estimate = sum(len(enc.encode(h.body)) for h in hunks)   # 115,000
# what the provider actually counts, on the Java repo:
#   hunk bodies         115,000
#   path + line headers   8,200
#   fences and labels     3,400
#   per-file instruction  6,000
#   ------------------------------
#   total               132,600   -> rejected against a 128,000 window

That total is an addition, and the four terms are worth reading as four different kinds of blindness rather than as one number. The 115,000 is the only term the estimator computed. The 8,200 of path and line-number headers is text the renderer adds per hunk. The 3,400 of fences and labels is text the renderer adds per block. The 6,000 of per-file instruction is text somebody wrote once and forgot was there. Added to the bodies they come to 132,600, which is four thousand six hundred over the 128,000 window — an overshoot smaller than the headers alone and smaller than the per-file instruction alone, so accounting for either one of them would have been enough on its own. Neither was accounted for, because neither was ever handed to the tokenizer.

Neither failure is fixed by adjusting the number 90. Lowering the fill target to 70 percent makes the Java repository stop failing and makes the Korean repository waste 53 percent of the window instead of 40, and it still breaks on the first review where one hunk is a 6,000-line generated file.

The 53 falls out of the same division as before. A count that runs half as high again as the truth turns a fill target into a real fill of that target divided by one and a half, so a 70 percent target is a real fill of a little under 47 percent, and what is left empty is a little over 53. Moving the target down therefore moves the Korean repository's waste up by exactly the amount it moves the Java repository's risk down, which is the signature of a single knob being asked to serve two content classes whose residuals have opposite signs. There is no setting of that knob that is right for both, because the quantity it is trying to control is not the same quantity in the two repositories.

A ratio is a planning number#

Recording a chars-per-token ratio per content class, as topic 1.3 recommends for cost forecasting, is a legitimate instrument: it tells you what a body of documents costs to process, what a market rollout does to a monthly bill, and roughly how many source files fit in a window. It is the wrong instrument for a boundary decision, because a boundary decision is about one specific request, and one specific request mixes classes in proportions you did not choose — a support ticket is English prose and a stack trace and a JSON payload the customer pasted, in a ratio that varies per ticket.

The tempting repair is a smarter estimator: detect the language, detect fenced code, apply a different divisor to each region. That is more code, more test surface and a better mean error, and it does not change the shape of the problem, because it still produces a number that is not the number the provider will compute. It fails on mixed content, on content whose class is ambiguous, and on the item outside every class you enumerated. The only thing that removes a residual is measurement of the thing being sent.

Characters per token is a summary statistic of a body of text, and a budget check is a claim about one request. Those are different kinds of object, and every failure in this chapter comes from using the first where the second was required — a divisor calibrated on English prose, a fill target chosen once, a margin sized against a mean. What the six-row error table shows is that the quantity you would need to bound is not bounded: it ranges from +5 percent to −77 percent across classes that arrive in the same product, and a single pasted attachment can leave that range entirely.

That leaves a limit on what measurement alone achieves, which the code-review bot demonstrates and the next chapter takes up. The Java repository was rejected while using a genuine tokenizer, correctly applied to the text it was given, because the payload contained material the packer never handed to the tokenizer. Everything that surrounds your text on the way to the model — the framing on each message, the system block, and above all the tool definitions with their schemas — is counted by the provider and is invisible to a function that only ever sees the strings you assembled.