Language and serialisation format

Measure one sentence in English and in Spanish, then the same 500 records as JSON and as CSV, and the counts separate by a factor of two to four rather than by a rounding error. Repeated quoted keys and nesting punctuation are billed once per record while a CSV header is billed once in total, and scripts outside the tokenizer's fitting mix fragment into more pieces per character, which is the Japanese and Thai case. This chapter gives both measurements, the mechanisms behind them, what the cheapest format costs back in reliability, and how to run the comparison so the numbers mean something.

Tokenization — read the current version at https://daviddominguez.dev/learn/en/tokenization/language-and-serialisation-format

Back to top