Character-level tasks

Topic 1.1 established what a request contains: bytes, then the pieces the fitted merges produce, then the integers indexing those pieces in a vocabulary. Nothing on that path is a character and nothing on it is a word. A model therefore never receives the string an engineer typed. It receives one integer per chunk, for whatever chunks the merge rules happened to carve out of that string, and every subsequent layer works on those integers.

That is a fact about representation, and its consequences are practical. Four kinds of work an engineer already trusts to succeed depend on detail that the chunking discards: arithmetic over digit strings, anything that inspects individual letters, reading a numeral whose punctuation follows a convention other than the writer's own, and carrying the same meaning in a language the vocabulary was not fitted on. How the merge rules were counted and frozen belongs to 1.1. What their output does to those four kinds of work is the subject here.

Consider one engineer who meets all four. They run a document pipeline for an accounts-payable product: it takes supplier invoices, extracts each line item and the grand total, and writes the result to a ledger, where a wrong number is money rather than a typo. The same platform team also runs a support-chat product in several markets, and the same team is on the hook for stripping card numbers out of anything a user types before it leaves the building. Every tokenization shown below was produced by running the exact string shown through cl100k_base, the tokenizer behind GPT-4, unless a second tokenizer is named beside it.

The invoice grand-total failure#

The first place tokenization surfaces is arithmetic, and it surfaces without warning. A supplier changed accounting systems, and the invoices it sends now write amounts with thousands separators where they used to write plain digit runs. Nothing else about the invoices moved. The pipeline's extraction prompt did not change, the model did not change, and the grand totals started coming back wrong. Both runs carry the same instruction — sum the line-item totals below and return the grand total, return the number only — and the same four amounts:

Read the last row against the four above it: only the separators differ, and the answer moves by a hundred units. Any accountant handed the plain invoice and the grouped invoice would say they are the same invoice, because the quantities are identical and the separators are decoration a reader's eye strips out without noticing. The extraction prompt did not get harder between one run and the other. The grand total moved anyway.

That is the shape of the whole subject, and it recurs in four places worth having named together before any of them is explained. A run of digits is cut into chunks whose edges have nothing to do with ones, tens and hundreds, so the arithmetic runs over pieces that do not correspond to the number's structure. A word's individual letters are absent from what the model receives at all, so a task that inspects them is asking about something that never arrived. The punctuation used to group a numeral varies by locale, and changing it moves the digit chunks underneath, which is what turned this supplier's change of accounting system into a ledger problem. And the same meaning written in different languages consumes wildly different numbers of tokens, which shows up on a bill and inside a budget rather than in an answer.

The first of those and the second look like different subjects, and they are the same one seen from two sides. A tokenizer's merge rules were fitted to compress whatever strings were frequent in a corpus. Frequency is the entire criterion. When a frequent chunk happens to span a boundary that a task cares about — the boundary between the hundreds digit and the tens digit, or the boundary between one letter and the next — the chunk swallows the boundary, and the model is handed an integer standing for the whole span with no seam in it anywhere. Digits lose place value that way and words lose letter identity that way, by the same mechanism, for the same reason, at the same moment in the pipeline. Neither cut has anything to do with how well the model reasons, and neither is repaired by making it reason harder.

The reason they belong in one chapter is that a reader who has met only one of them tends to assume the other works differently — that letter counting is a quirk of spelling while arithmetic is a matter of capability, or the reverse. They are one fact about the input wearing two costumes. From here they separate, because what they cost and what answers them are not the same.

Digit tokenization#

The first of those costs is digit tokenization: how a tokenizer's merge rules cut a run of digits into tokens: by whichever digit chunks were frequent enough in the training corpus to earn a vocabulary entry, without regard for how many digits the numeral has, where its place-value boundaries fall, or what punctuation groups it.

Run the plain invoice's amounts through the tokenizer and the cuts are visible.

482.50     ['482', '.', '50']
1205.00    ['120', '5', '.', '00']
3860.75    ['386', '0', '.', '75']
1284.50    ['128', '4', '.', '50']

1205.00 arrives as 120, then 5, then a decimal point, then 00. The token 120 spans the thousands digit, the hundreds digit and the tens digit of an amount whose thousands digit is the one that matters most; the units digit is left standing alone in a token of its own. 3860.75 is cut the same way at the same offset, leaving 386 and 0, so the trailing zero of three thousand eight hundred and sixty is a separate integer from the rest of it. Nothing in this arrangement corresponds to the number. The pieces are 123-sized because three-digit runs are the longest digit chunks this vocabulary holds, and the run is consumed left to right until it is exhausted.

Now the grouped invoice, printing only the amounts the supplier's new system rewrote. The line-item label column and the instruction are unchanged, and 482.50 is unchanged because there was nothing in it to group.

1,205.00   ['1', ',', '205', '.', '00']
3,860.75   ['3', ',', '860', '.', '75']
1,284.50   ['1', ',', '284', '.', '50']

The digit pieces are different pieces. 120 and 5 have become 1 and 205; 386 and 0 have become 3 and 860. Neither arrangement is the number's structure and neither is a mistake by the tokenizer, which is doing precisely what it was fitted to do. They are two different sequences of integers standing for one quantity, and the model was given one of them on Monday and the other on Tuesday.

This is a byte-pair merge process behaving exactly as 1.1 described it, and the tutorial at hundredblocks.github.io and Andrej Karpathy's Let's build the GPT Tokenizer both walk the same process over numerals in running code, where the arbitrariness is easier to feel than to be told. What the merges learned about digits is what the corpus happened to contain: strings of three digits are everywhere in web text, so three-digit entries were earned, and the entry for 120 exists for reasons that have nothing to do with the number one hundred and twenty ever being added to anything.

The measured consequence has a source. Counting Ability of Large Language Models and Impact of Tokenization (arxiv.org/pdf/2410.19730) is the paper that connects tokenization directly to counting and digit-task failure rather than treating those failures as a reasoning deficit. It holds the counting task fixed and varies only how the string is segmented, and reports that segmentation alone moves accuracy: the same counting problem, presented so that the units the task is about land in their own tokens, is answered substantially more reliably than the same problem presented as merged chunks. The paper's own framing is that reasoning procedures raise counting accuracy while tokenization sets a ceiling on what those procedures can reach, because a procedure cannot iterate over units it was not given.

Two other explanations arrive before the tokenizer does, and the evidence already on this page settles each.

The first does not hold: it is not the context window. The plain invoice and the grouped invoice are five lines apiece; the grouped one is longer by three characters, which is the three commas. Topic 1.4 owns the budget that a prompt can exhaust, and neither of these prompts is within several orders of magnitude of it. A capacity explanation for a five-line prompt is a capacity explanation for nothing.

Nor is it the model reading the commas as list separators that split one number into three. Take a bare seven-digit run and vary only its separator.

1234567     ['123', '456', '7']
1,234,567   ['1', ',', '234', ',', '567']
1.234.567   ['1', '.', '234', '.', '567']
1 234 567   ['1', ' ', '234', ' ', '567']

A period is what this reader writes a decimal point with, and a space is not a separator of anything at all, and each of them regroups the digits exactly as the comma does: 1, 234, 567 in every case. The comma is not being read; it is interrupting a digit run, and a period and a space interrupt one identically. The bare form settles it from the other direction, because it contains no punctuation whatsoever and still comes apart, into 123, 456 and 7 — three pieces whose edges land nowhere near a place-value boundary and nowhere near anything a reader wrote. Only the position of the cut changed across those four strings, and only the merge rules moved it.

What the invoice failure costs follows from what the model was handed. It was handed a sequence of integers and it produced a continuation, and the continuation that follows a familiar sequence of digit tokens is the one those tokens were followed by in training. That is pattern-matching over a token sequence rather than the execution of an addition, which is why reliability here rides on how familiar the specific sequence is rather than on how hard the sum is. 482 . 50 is a shape web text is saturated with. 120 5 . 00 is a shape it is not, and the arithmetic over it is correspondingly a guess made from further away.

The token-count heuristic#

A reader who now knows a numeral's segmentation is unpredictable wants a cheap way to tell, before running anything, which of the prompts in front of them is the riskier one. Counting is cheaper than measuring, and counting is what the available signal offers.

The token-count heuristic is the practice of ranking numeric prompts by how many tokens each numeral in them costs, and treating the more fragmented prompt as the more likely to produce a wrong number. It requires no model calls, runs in the time it takes to encode a string, and is right often enough to be worth having.

It is a heuristic and not a law, and the case that keeps it one is a numeral whose familiarity outruns its fragmentation. Compare a four-digit year against an arbitrary four-digit number of the same length under p50k_base, the tokenizer behind the GPT-3 family.

2019   ['2019']
4831   ['48', '31']
2026   ['20', '26']

2019 is one token. 4831 is two, and the year 2026 is also two, because p50k_base was fitted before that year existed in any quantity of text and its merges were frozen at that point. Familiarity is a property of a corpus at the moment it was counted, not a property of numbers, and a year is only cheap while it is a year the corpus was full of.

The same three strings under the tokenizer this chapter has been using tell it differently again.

2019   ['201', '9']
4831   ['483', '1']

Two tokens each. A count taken here cannot separate them at all, and yet 201 followed by 9 is a pair that occurs in text every time anyone writes a date in that year, while 483 followed by 1 is a pair almost nothing puts together. The year is the safer of the two for the reason the digit runs gave earlier — the continuation is drawn from what followed the sequence in training — despite being segmented no better. The heuristic's own signal is blind to the difference.

A blind spot of that kind is what sends an engineer looking for a second knob, and the one within reach governs the model's output rather than its input. It is worth saying what that knob governs before reaching for it. Token count and temperature both look like ways to make an unreliable numeric prompt behave, and both are settings an engineer controls before the request goes out. What they act on is not the same thing. Temperature governs how the model picks among candidate next tokens: turn it to zero and the same prompt returns the same answer every time, which makes an unreliable answer repeatable rather than making it right. The digit run was cut into tokens before generation began, by a step that has no temperature and no sampling in it. Token count is a claim about the input the model received. Temperature is a claim about how it chose its output. No setting of the second restores place value the first threw away.

So the heuristic is a first sort. It is worth running over a numeral-heavy prompt template before that template ships, it costs nothing, and a prompt whose numerals fragment into many pieces deserves a look before a prompt whose numerals do not. What it cannot do is tell an engineer that a particular number will come back right, because the thing that decides that is the familiarity of a specific sequence and the count does not measure familiarity.

Character blindness#

Everything so far has been a run of digits cut into the wrong-sized pieces. The pieces were at least made of the digits. Turn to a task that needs individual letters, and the same chunking removes the material rather than rearranging it, because a chunk several letters wide arrives as one integer with no letters in it.

Character blindness is a model's inability to reliably count, reorder or inspect the individual letters of a word, caused by the letters never being inputs: what arrives is one token ID per chunk the merge rules assigned, and a chunk is frequently the whole word.

The pipeline team hit it on a task nobody would predict a model could fail. An internal tool checks HTTP header names for a common configuration mistake, and asked the model to count a letter in one.

How many times does the letter i appear in the header name Authorization?

3

There are two. What the model was given explains the answer better than any account of its reasoning does.

Authorization   ->   ['Authorization']

The header name that produced that answer is thirteen letters, and it arrives as one token, one integer. The word is common enough in web text — every API document, every request log, every code sample — that the merges welded it into a single vocabulary entry, and that entry is exactly as decomposable as the entry for refund or the entry for a comma. Nothing in the forward pass takes it apart. Whatever the model knows about the letters in that word it knows the way it knows a capital city: from text that stated it, not from the input in front of it.

This is the failure Chris Hay's walkthrough how the tokenizer for gpt-4 (tiktoken) works and why it can't reverse strings demonstrates by picking words whose tokenization lands nowhere near individual letters and watching character-level requests come back wrong, and it is the one Karpathy's lecture closes on when it enumerates what tokenization costs. The demonstration is always the same in outline: choose a word the vocabulary holds whole or nearly whole, ask something about its letters, and the answer is a plausible number near the true one.

Three explanations get reached for before this one, and each predicts something that does not happen.

The first is that attention is failing to look far enough back, which predicts that the failure depends on distance, and the word sits in the same sentence as the question. There is nothing to look back past.

The second is that the tokenizer is stripping or normalising something, which predicts that the text does not survive the round trip, and it does: decode the token IDs and Authorization comes back character for character, capital A and all. Nothing was removed. The thirteen letters are all still recoverable from the integer; they are simply not separately present in it, and a decoder reconstructs them while the model's layers never see them apart.

The third is the one a reader assumes without noticing they assumed it: that the model was not trained on enough spelling. Spelled-out words are ordinary web text. Acronym expansions, hyphenation guides, crossword clues, letter games, style guides arguing about doubled consonants — a training corpus is thick with text that names letters one at a time. The same model that miscounts the letter in that header name will define the word, use it correctly in a sentence about an API, and recite the rule that governs its spelling, because all of that is knowledge that arrived as text. The gap is not in what the model learned and it is not in what it can do with what it learned; it is upstream of both, at the point where thirteen letters became one integer.

Character-level tasks#

Letters being invisible matters exactly as much as a task depends on them, so the useful thing to carry away is not the failure but the ability to recognise, before writing a prompt, that a task is in the affected class.

A character-level task is a task whose correct answer requires manipulating or inspecting individual characters rather than the meaning of the text they spell.

The header-name count is one. Reversing a string is one. Naming a word's fourth letter is one. Holding a generated identifier to exactly eight characters is one, and it is the kind that hides, because the requirement arrives worded as a product constraint rather than as a character operation. Checking whether two account references differ only in their last four characters is one. And the example worth carrying through the rest of this chapter is one: deciding whether a sixteen-digit card number appears at some exact position in a block of text a user typed, so that the characters at that position can be removed before the block goes anywhere.

That last belongs to a discipline of its own, and topic 14.6 owns PII detection and redaction as a subject. What it contributes here is a character-level task with a consequence attached, which is what makes it the right one to return to once the question becomes what to do about any of this.

Before that question can be answered, though, the distinction worth holding on to is between character blindness and a character-level task, because they are easy to fuse and they behave differently. What they share is the gap: the letters the merge rules hid inside a token. Character blindness is a property of the model, constant across every request, present whether or not it is being exercised. A character-level task is a property of the work, and it is what turns that constant into a wrong answer. A pipeline that never asks about characters runs into character blindness on every request and is never harmed by it. Recognising the task is therefore the operative skill, because the model's side of it is not going to change.

Character spacing#

The header-name count failed on an input the model could not decompose, and the decomposition is a thing the prompt can do for it. That is worth trying before anything architectural, because it costs a string operation.

Character spacing is the practice of inserting separators between a word's letters in the prompt, so that more of those letters land in tokens of their own instead of inside one chunk.

Here is the header-name count as the tool sent it, and as it sends it now.

How many times does the letter i appear in the header name Authorization?

3

How many times does the letter i appear in the header name
A u t h o r i z a t i o n ?

2

What changed between them is visible in the encoding.

A u t h o r i z a t i o n
->  ['A', ' u', ' t', ' h', ' o', ' r', ' i', ' z', ' a', ' t', ' i', ' o', ' n']

Thirteen tokens where there was one, and every letter now has an integer to itself. The question asks about i, and there are now two tokens in the sequence that are an i, sitting where they sit, available to be attended to like any other token in the prompt. The task stopped being a question about a word and became a question about a sequence the model was actually given.

Chain-of-thought, which topic 4.4 owns, reaches the same place by the other route, and what it shares with character spacing is worth putting first because a reader who has met one will otherwise fold in the other. Both work by getting individual characters into the token stream, where subsequent tokens can attend to them; both replace a question about a chunk with a question about a sequence; and both are supplying an input rather than improving a computation. Where they part is when the characters arrive. Character spacing puts them there before the model runs, by rewriting the prompt string in code. Chain-of-thought puts them there during generation, by having the model emit them and then attend to what it emitted. The prompt is the author's to change in one case and the model's output is doing the work in the other, and those vary independently: a prompt can be spaced without asking for any enumeration, and an enumeration can be asked for on a prompt nobody spaced.

Spell out the header name Authorization one letter at a time, then count how many
of those letters are i.

A - u - t - h - o - r - i - z - a - t - i - o - n
2

Reversal is the case a reader tests next, and it moves the same way and further, because reversal needs every character rather than a count of one of them.

Reverse the header name Authorization. Return the reversed string only.

noitazirothuA

Spell out the header name Authorization one letter at a time, then write those
letters in reverse order. Return the reversed string only.

A - u - t - h - o - r - i - z - a - t - i - o - n
noitazirohtuA

The direct answer has t and h the wrong way round in the middle, which is what a plausible guess about a thirteen-letter word looks like when the letters were never inspected. The enumerated answer is the string. That result runs against an intuition worth naming where it appears, because the enumerated answer is longer, and a longer output does ordinarily accumulate more chances to go wrong. Here the extra length is the enumeration, and the enumeration is the missing input arriving late rather than additional opportunity for error. Length moved in the helpful direction because of what the added tokens were.

Both techniques stop at the same line, and the line is what they cannot buy. Putting characters into the token stream supplies what was missing, and it does not convert a character-level task into a deterministic operation. The enumeration is itself produced by the model and can come out wrong, in which case everything downstream of it is wrong with confidence. Spacing lengthens the prompt and enumerating lengthens the output, and both are billed. What comes back is a probabilistic pass over tokens standing in for something a line of code returns exactly, every time, at no cost.

Whether a task the reader has not been shown is even on the helped side of that line has a test with one question in it: does the answer depend on something the merge rules hid inside a token, or on something already sitting in the token stream? Work it in both directions. Counting the letter in that header name is on the hidden side, which is exactly why spacing and enumerating move it — each supplies an input that was absent.

Counting the words in a sentence is not.

The refund was approved by the finance team on Tuesday morning.
->  ['The', ' refund', ' was', ' approved', ' by', ' the', ' finance', ' team',
     ' on', ' Tuesday', ' morning', '.']

Eleven words and twelve tokens, and every word after the first begins a token of its own carrying the space that preceded it, for the reason 1.1 gave for leading-space entries existing at all. Word boundaries survive this tokenizer largely intact. The model can already see where each word starts, so a wrong word count is a running tally being lost over material that was supplied, not a failure to see the material. Restating that sentence one word per line adds no input to it. Chain-of-thought therefore moves word counting far less than it moves letter counting, and whatever improvement it does give comes from the tally being written down where the model can attend to it rather than from characters arriving that had not arrived before.

Digit grouping#

The same reading problem appears in numbers, where the pieces a numeral breaks into depend on how it was punctuated. The supplier that changed accounting systems wrote the same amounts a different way, and the pipeline's other suppliers are spread across several countries, which means the same quantity can arrive written in more than one arrangement before anybody has changed what it means.

Digit grouping is the formatting convention that inserts separator characters into a numeral at fixed intervals, with the separator character and the interval both varying by locale — the reader's own 1,234,567 sitting beside a supplier's 1.234.567 or 1 234 567, all naming one quantity.

The decimal mark varies with it, and the pair of choices is what makes this a hazard rather than a nuisance. A supplier writing in a convention that groups with a space and marks the decimal with a comma sends 1 284,50 where the pipeline's own documents say 1,284.50. The extraction prompt reads the pipeline's own convention correctly on almost every document it sees. On this supplier's documents it starts missing, and the totals it returns are not obviously malformed, which is how the drop reaches a ledger before it reaches a dashboard.

1,284.50   ['1', ',', '284', '.', '50']
1 284,50   ['1', ' ', '284', ',', '50']
1.284,50   ['1', '.', '284', ',', '50']
1284.50    ['128', '4', '.', '50']

Identical value, four sequences. The first three keep the digit pieces the same and swap the punctuation tokens between them, which is a smaller difference than it looks and a real one: , and . and are different integers, and the sequence a continuation is drawn from is different in each. The last is the ungrouped form the pipeline used to receive, and its digit pieces are not the others' digit pieces at all. There is no canonical numeral in this system, only whichever arrangement the supplier's accounting software emitted, tokenized as it arrived.

Locale-aware number parsing#

Since one value can arrive in any of those arrangements, a system that reads numerals out of documents needs a way to interpret one correctly whichever convention produced it, instead of assuming its own.

Digit grouping and the interpretation of a grouped numeral act on one object, the numeral string, from opposite ends. Digit grouping is what produces the variants; it happens in the supplier's accounting software, before anything of the pipeline's has seen the document. Locale-aware number parsing is interpreting a numeral according to the punctuation conventions of the locale it came from, rather than one assumed decimal mark and one assumed grouping separator, and it happens in the pipeline, after.

Which of those two ends a drop in extraction accuracy came from is the thing the debugging has to establish, and it is not established by reading the prompt. Nobody debugging the drop on that supplier's documents should touch the prompt first, because the prompt is the expensive place to look and the last place the evidence arrives. Three measurements come first, and each of them takes minutes.

Collect the failing documents into a set of real examples rather than reasoning from one. A drop that shows up on a supplier is a drop on a cohort, and the cohort is what any subsequent number has to be computed over.

Run the supplier's numeral strings and the pipeline's own numeral strings through the tokenizer the request actually uses, and compare the boundaries. That is the exhibit above, produced on the pipeline's own documents rather than read out of a chapter, and it answers a question that decides everything after it: whether the segmentation moved with the grouping convention, or whether something else in the extraction pipeline — a different page layout, a different OCR pass, a different field the totals now sit in — moved at the same time the supplier did.

Classify what the model returns on the failures rather than counting them. A returned value that has lost its decimal mark, one that has kept the digits and moved the mark, and one that is structurally malformed point at different causes, and the classification is what stops a fix from being aimed at the wrong one.

Those three say what is failing and why. One more is needed before anything is built, and it is about the fix rather than the fault: record the current accuracy on that supplier's format before any change ships. Without it, whatever ships next is defended by an argument rather than by a number, and the argument will be that the prompt reads better now.

Two fixes are then available. The prompt-side fix is a handful of few-shot examples written in the supplier's exact format, so that the arrangement the model is being asked to read is an arrangement it has just been shown being read. It is quick, it needs no deployment beyond a template change, and it depends on the model generalising from those examples to every numeral in that format it has not been shown, which is the property that failed in the first place.

The system-side fix normalises the numeral to one canonical arrangement in code, before the document reaches the prompt at all, so the model only ever receives the arrangement it was already reliable on. It requires knowing or inferring the supplier's locale, and it requires a normaliser that refuses what it does not recognise instead of guessing, because a normaliser that silently mangles an unfamiliar arrangement is worse than the model was.

The system-side fix is the one to ship. It is deterministic, so it returns the same canonical string for the same input on every run and can be unit-tested against the collected examples to an exact answer rather than to a rate. It removes the variable rather than compensating for it, which means it does not have to keep being right as model versions change underneath it. The few-shot examples ride along as a second layer, where they cost a few dozen tokens and cover documents whose locale the normaliser could not establish.

Token inflation#

Digit grouping is one constraint the tokenizer imposes on content it otherwise serves reasonably well. The question underneath the support-chat product is a larger one: what happens when the tokenizer does not serve the content's language at all. That is a question about the whole shape of a piece of text rather than about anything inside it.

Token inflation is the ratio between the tokens a piece of text consumes and the tokens an equivalent-meaning piece of text consumes in the tokenizer's best-supported language, which for the hosted vocabularies in general use is English.

The support-chat team is opening in Spain and Japan, and the product manager wants one price per seat across both markets on the grounds that the model is multilingual and therefore works the same. Here is one customer message, as it arrives in each market.

I was charged twice for my subscription this month and I would like the second
charge refunded to the card I paid with.

Me han cobrado dos veces la suscripción este mes y me gustaría que el segundo
cargo se devolviera a la tarjeta con la que pagué.

今月サブスクリプションの料金が二重に請求されました。二回目の請求を支払いに使用したカードへ返金していただきたいです。

English    119 characters   119 bytes   24 tokens   1.00x
Spanish    128 characters   131 bytes   34 tokens   1.42x
Japanese    58 characters   174 bytes   56 tokens   2.33x

The Japanese message is the shortest of the three by more than half when counted in characters and the most expensive by a factor of two and a third when counted in the unit that gets billed. Its pieces show why.

['今', '月', 'サ', 'ブ', 'ス', 'ク', 'リ', 'プ', 'シ', 'ョ', 'ン', 'の', '料', '金', 'が', ...]

Almost every character is its own token, and several are worse than that: a few of these characters do not survive as one entry at all, and their UTF-8 bytes are split across more than one, which is how fifty-eight characters reach fifty-six tokens against an English line that packs nearly five characters into each. Spanish sits between, and its inflation comes from a different place — suscripción breaking into sus, cri, pción and devolviera into dev, olv, iera, ordinary words fragmenting because the corpus the merges were counted on held fewer of them.

Santhosh Thottingal's The Broken Token: Tokenization for Malayalam Language Models reports the same effect at the far end of the range, on a language whose script the mainstream vocabularies cover poorly, and the value of that account is that it measures rather than characterises: it walks through what a real corpus in that language costs under tokenizers built elsewhere, and shows the fragmentation reaching down to individual characters and below them to bytes. Redis's Tokenization in LLMs: What AI App Devs Need to Know sets out the arithmetic a developer runs on top of a ratio like that, which is the same arithmetic in every case: tokens per unit of work, multiplied by units of work, multiplied by a per-token price that does not vary by language.

That last clause is the whole of the cost argument, and it needs separating from a question it gets confused with. Answer quality and token inflation are two consequences that can be read off the same fragmentation, and they are measured differently and can move independently. The Japanese message above is answered correctly: the model understands that a subscription was billed twice, that the customer wants the second charge back, and that the refund should go to the original card, and a support agent reading the reply would find nothing to fix. Quality held. The request still cost 56 tokens against 24, and still occupied 56 tokens of whatever budget the conversation history is allowed. Cost and capacity moved while quality did not, which is why a capability answer — the model handles Japanese fine — does not address the question and why a quality evaluation will never surface the problem.

What those two consequences belong to is owned elsewhere and worth naming rather than re-deriving. Topic 1.3 owns cost per language, which is the per-language cost difference this ratio drives. Topic 1.4 owns the context window, which is the budget that shrinks, in token terms, for the same quantity of meaning. Topic 1.8 owns truncation, which is the failure a shrunken budget raises the odds of hitting without announcing itself.

Converted into what a pricing decision needs, the ratio says two things. The model line of the cost of serving a Japanese conversation is 2.33 times the English one at identical content, because the price is per token and the token count is what moved. And a conversation history capped at a fixed token budget holds fewer turns in Japanese than in English by the same factor: if the team's cap admits forty turns of an English conversation, it admits roughly twenty-eight of the Spanish one and seventeen of the Japanese one before the oldest turns start falling out. The second of those is a product behaviour difference rather than a cost one. The Japanese customer's agent forgets the start of the conversation sooner, the support metrics for that market diverge, and the divergence gets attributed to the model rather than to the budget.

The answer to the product manager is that number rather than an objection. Identical pricing across those markets assumes identical token consumption per conversation, and the measurement above shows that assumption is false by a factor of more than two on the Japanese side; a Japanese seat sold at the Spanish price therefore carries a materially thinner gross margin and reaches its history limit in fewer turns, and neither of those is visible in a quality review. The counter-proposal is a measurement rather than a price: take the team's own conversations, translate a real sample into each supported language, run them through the tokenizer the team actually ships against, and price or budget on measured tokens per conversation instead of on seats or message counts. And if the business needs one flat price regardless — which is a legitimate call, since price differentiation by language reads badly and simplicity is worth something — set it from the highest-inflation language in the supported set, and record the gap as a cross-subsidy the team measured and chose rather than one it finds later in a margin report.

Deciding when to route a task to code#

Knowing what a task costs in one language or another still leaves the question underneath it unasked: whether a prompt should be running that task at all. Three questions settle the one the pricing arithmetic cannot, and the order they are asked in is what routes the task.

One path out of the tree ends in code. Every other path ends in a prompt that has to be measured, split, or fed its characters deliberately, which is why the first two questions are asked before the third rather than after it.

Walk the card-number redaction down the tree. The answer is a block of text with certain characters removed, and moving one character of a card number leaves a card number partly intact, so it is character-exact. A miss sends a card number to wherever the block was going, which is a compliance event and not a quality complaint, so it is intolerant. And the pattern is sixteen digits, optionally grouped in fours by spaces or dashes, which is expressible.

const CARD = /\b(?:\d[ -]?){15}\d\b/g;

'card 4111111111111111 on file'      -> ['4111111111111111']
'card 4111 1111 1111 1111 on file'   -> ['4111 1111 1111 1111']
'card 4111-1111-1111-1111 on file'   -> ['4111-1111-1111-1111']
'mixed 4111 1111-1111 1111 here'     -> ['4111 1111-1111 1111']
'seventeen 41111111111111111 digits' -> null
'fifteen 411111111111111 digits'     -> null
'phone 555-0142 and total 1,284.50'  -> null

Sixteen digits with an optional space or dash after each of the first fifteen, bounded at each end so that a longer digit run is not clipped down to a sixteen-digit match inside it. Exact on the pattern it describes, and exact on the same text a year from now, it still does not decide whether a matched string is a card number. It performs no Luhn check, so a sixteen-digit order reference matches it exactly as a card does, and a real deployment has to decide what happens to those — redact them too and lose a reference, or add the check and accept whatever it rejects. That gap is a decision to make rather than a defect to fix, and making it deliberately is available here in a way it is not available inside a prompt.

Routing this to code rather than to a better prompt is an instance of a discipline topic 5.7 owns as verifier-first design, and the judgement it encodes is the reason the alternatives fail rather than a preference about tooling.

A self-check pass — the model redacts, then the model verifies its own redaction — reads like defence in depth and is not, because the verifier receives the same block of text through the same tokenizer and is therefore missing exactly what the first pass was missing. Two passes over an absent input is one blind spot consulted twice.

A few-shot prompt with worked redaction examples is quick in the way few-shot is always quick, and fails where it always fails — on the unusual inputs nobody put in the demo. For a control whose job is to never emit a card number, most of the time is the exact property that disqualifies it, and the failures arrive under load and under unusual formatting rather than in review.

A fine-tuned model is the third reach, and the one with a budget behind it. Training changes the weights and leaves the input representation alone, so what improves is the model's guess about a block of text it still receives as chunks, with the digits still inside them. It costs a training run and a serving path and an ongoing evaluation commitment, and at the end of it the digits are still not separately present. It is the expensive version of the same missing input.

The general form is the rule the card number demonstrates. Route a task to code when its answer is character-exact and expressible as a deterministic pattern, and keep the model for the parts that are genuinely semantic — which document this is, which field is the total, whether this sentence is a complaint. That division is a triage signal and a piece of domain judgement rather than a formula that decides itself, the same way the token-count heuristic ranks prompts without settling any of them.

Drawing that division at all is possible because the chapter now has names for what sits on either side of it. Tokenization changes a task in four places, and each of them has a name the reader can now use. Digit tokenization cuts a numeral into chunks that ignore place value, which is why an invoice's grand total can move when only its formatting did. Character blindness removes individual letters from what the model receives, which is why counting them is a question about something that never arrived. Digit grouping produces the numeral variants that make the first of those a live problem across suppliers and locales, and locale-aware number parsing is what a system does about them. Token inflation measures how much more of the billed and budgeted unit the same meaning costs in one language than another.

Three things respond to them. Character spacing puts the characters into the prompt. Chain-of-thought puts them into the output. The token-count heuristic ranks prompts by fragmentation before either is needed, at no cost and with no guarantee. What none of them does is make a character-level task exact, which is why the decision that matters most is the one about which tasks belong in code.

How much any of this matters is proportional to how character-exact, numerically fragile or non-English-heavy the work is. A summarisation task over English prose tolerates ordinary token noise and will never notice a word of this chapter. A redaction control, a totals extractor and a multilingual per-seat price sit at the other end, and the distance between those ends is large enough that no general rule covers both. Where a specific prompt or a specific language pair falls on it is not a thing to be reasoned out in advance. It is a measurement, taken with the tokenizer the request will actually use, on the content that will actually arrive.