Whitespace and prompt seams

'hello' and ' hello' encode to two different integers, because modern BPE vocabularies fold the preceding space into the word piece. The consequence is practical rather than academic: few-shot delimiters, stop sequences, aligned column padding and naive string concatenation at prompt-assembly time all change the token stream without changing anything a human reader would notice, and counting the fragments separately does not give the count of the assembled prompt. A prompt that looks identical in a diff can tokenize differently, so the assertion worth writing is on the token IDs.

Tokenization — read the current version at https://daviddominguez.dev/learn/en/tokenization/whitespace-and-prompt-seams

Back to top