The four-characters-per-token heuristic

The rule of thumb is good enough to size English prose within about twenty percent, and it collapses on minified JSON with quoted keys and deep nesting, on pasted stack traces, and on scripts the vocabulary barely covers. A context-budget check written as len(text) // 4 is a bug waiting for its first Thai document, because every content class except English understates and Thai inverts the relationship entirely. This chapter gives the ratio by content class, shows why lowering the divisor is not a repair, and sets out what measuring actually costs and where estimating is still the right tool.

Tokenization — read the current version at https://daviddominguez.dev/learn/en/tokenization/the-four-characters-per-token-heuristic

Back to top