Byte pair encoding and rare strings

A vocabulary is fitted to a corpus by byte pair encoding, so 'policy' survives as one entry while a surname like 'Szczepanski' is reassembled from five fragments. Run words and then a whole paragraph through a real tokenizer library and read the pieces back rather than guessing. Seeing which strings survive intact and which shatter explains why unusual names, product codes, UUIDs and encoded blobs cost far more than their length suggests, and why the spread between the cheapest and most expensive content in one prompt is close to five to one.

Tokenization — read the current version at https://daviddominguez.dev/learn/en/tokenization/byte-pair-encoding-and-rare-strings

Back to top