Token ratios by content class

Five hundred words of Python does not cost what five hundred words of English costs: indentation, punctuation clusters, split identifiers and repeated syntax fragment into more pieces, and a script the vocabulary covers poorly fragments further still. This chapter measures characters per token for six content classes, converts each into a price per thousand requests and a context-window capacity, gives the ten-line measurement and the continuous-integration guard that stops the ratio drifting, and separates the repairs that work from the ones that look like they should.

Token economics — read the current version at https://daviddominguez.dev/learn/en/token-economics/token-ratios-by-content-class

Back to top