Digit grouping and number formats

'1284.50' and '1 284,50' are the same quantity and different token sequences, and extraction accuracy moves with the sequence rather than with the value. This chapter encodes six real money formats, measures exact-match extraction on each over 200 documents, classifies the wrong answers to show they are re-groupings of the same digit tokens rather than misunderstood conventions, and repeats the measurement on dates. It also marks the boundary: token count is not the operative variable, and the numbers are a property of one vocabulary rather than a law.

What tokenization actually breaks — read the current version at https://daviddominguez.dev/learn/en/what-tokenization-actually-breaks/digit-grouping-and-number-formats

Back to top