Tokenization

Read these first

• What a language model is

What you will know

• Run a piece of text through a real tokenizer and inspect the resulting tokens

  Use a real tokenizer library rather than guessing. Seeing the boundaries once fixes a dozen later misconceptions.

• Estimate the token count of a paragraph within ~20% before measuring it

  A rough rule holds for English prose, but you should know when it breaks — code, JSON, and other languages all deviate.

• Show a word that splits into several tokens and one that doesn't

  Common words are one token; rare or compound ones split into several. This is why unusual names cost more than you expect.

• Compare token counts for equivalent English, Spanish, and JSON inputs and account for the differences

  Language coverage and syntax regularity affect compression, while punctuation and quoted keys can make structured text especially expensive.

• Trace a tokenized input from text through token IDs into a model request

  The model's learned computation consumes vocabulary indices, often after byte-level preprocessing, so character boundaries are not native inputs.

Chapters

1. 1

   From text to token IDs

   Follow the string 'Refund policy' through the encoder: UTF-8 bytes, greedy merges into subword pieces, a lookup that turns each piece into an integer vocabulary index, and finally a row of the embedding table. Everything the model computes starts at those integers, which is the mechanical reason characters are not something it can inspect and the reason a token count is a property of a specific vocabulary rather than of your text. The chapter then follows the same trace across the API boundary: a hosted request carries your string, tokenised on the provider's side with the provider's own vocabulary, and the resulting counts come back in the response's usage fields.

   Read chapter
2. 2

   Byte pair encoding and rare strings

   A vocabulary is fitted to a corpus by byte pair encoding, so 'policy' survives as one entry while a surname like 'Szczepanski' is reassembled from five fragments. Run words and then a whole paragraph through a real tokenizer library and read the pieces back rather than guessing. Seeing which strings survive intact and which shatter explains why unusual names, product codes, UUIDs and encoded blobs cost far more than their length suggests, and why the spread between the cheapest and most expensive content in one prompt is close to five to one.

   Read chapter
3. 3

   Whitespace and prompt seams

   'hello' and ' hello' encode to two different integers, because modern BPE vocabularies fold the preceding space into the word piece. The consequence is practical rather than academic: few-shot delimiters, stop sequences, aligned column padding and naive string concatenation at prompt-assembly time all change the token stream without changing anything a human reader would notice, and counting the fragments separately does not give the count of the assembled prompt. A prompt that looks identical in a diff can tokenize differently, so the assertion worth writing is on the token IDs.

   Read chapter
4. 4

   The four-characters-per-token heuristic

   The rule of thumb is good enough to size English prose within about twenty percent, and it collapses on minified JSON with quoted keys and deep nesting, on pasted stack traces, and on scripts the vocabulary barely covers. A context-budget check written as len(text) // 4 is a bug waiting for its first Thai document, because every content class except English understates and Thai inverts the relationship entirely. This chapter gives the ratio by content class, shows why lowering the divisor is not a repair, and sets out what measuring actually costs and where estimating is still the right tool.

   Read chapter
5. 5

   Language and serialisation format

   Measure one sentence in English and in Spanish, then the same 500 records as JSON and as CSV, and the counts separate by a factor of two to four rather than by a rounding error. Repeated quoted keys and nesting punctuation are billed once per record while a CSV header is billed once in total, and scripts outside the tokenizer's fitting mix fragment into more pieces per character, which is the Japanese and Thai case. This chapter gives both measurements, the mechanisms behind them, what the cheapest format costs back in reliability, and how to run the comparison so the numbers mean something.

   Read chapter
6. 6

   Token counts across model families

   A per-document price built with today's model's tokenizer measures that tokenizer, not your corpus. Another provider ships a different vocabulary and different merge rules, so the same documents produce counts that span 38 percent, and a flat percentage buffer assumes an error far more stable than it really is. What carries over is the sampled corpus and the counting method, so this chapter covers how to take that measurement on a model you have not integrated: offline against a published vocabulary, through a count_tokens endpoint, or by sending a sample as real calls and reading the counts back. It closes on the exposure no input measurement bounds, which is output.

   Read chapter