The limit is counted in the model's tokens, not your characters

A retrieval system cannot embed a long document whole. Chunking is the step that splits every document into passages small enough for the model to encode, stores one vector per passage, and searches over those vectors, because a 40-page manual has no chance of fitting into a single call. Choosing how large a passage may be means choosing a number, and that number is almost always written in characters, since characters are what a string-splitting library counts. The model's ceiling is not written in characters. It is written in tokens, and the exchange rate between the two units is a property of the text rather than a constant, which is what turns a character rule into a bet.

A token is one of the pieces a model's tokenizer cuts the input into before the model sees it — mostly whole common words, with fragments of words standing in for anything the vocabulary does not hold. The word information costs one token, because it is a word the vocabulary has an entry for. kubectl costs four, because it is not a word at all and has to be spelled out of fragments. Every limit in this chapter is counted in those pieces, and the number of pieces a 2,000-character passage produces is exactly the quantity a character rule is guessing at.

Where the ceiling comes from#

Every embedding model has a maximum sequence length, the largest number of tokens it can encode in a single call, and for bge-base-en-v1.5 that number is 512. The ceiling follows from how the encoder was built rather than from a policy somebody set, and the chain is short enough to check link by link. A transformer encoder processes all of its input positions at once and has no built-in notion of which token came first, so position has to be supplied to it as data: each slot in the sequence has its own learned position vector, added to the token sitting in that slot. Those position vectors are parameters, which means a fixed, finite number of them is baked into the weights file. A token arriving in slot 513 at a model that holds 512 position vectors has no position vector to be given, and there is nothing the model could do to invent one.

That is why no amount of paying more, waiting longer, or setting a flag raises the number. It is a count of parameters that physically exist in the checkpoint, and it is fixed in the published model rather than settable per request. What happens at the ceiling is a different kind of fact, and it is worth separating from the ceiling itself, because it is a decision rather than a constraint. Almost every client library truncates: it keeps the first 512 tokens, drops everything after them, and returns an ordinary-looking vector with no exception, no warning, and no field in the response recording that anything was removed. Refusing an over-long input would have been an equally legal thing to build, and the word almost in that sentence is where the few interfaces that refuse live. Truncating is what the common ones chose, which is why the failure arrives as a plausible vector rather than as an error.

The discarded tail is not stored anywhere and is not searchable, and no later change to the query or the prompt recovers it, because it never reached the model. Chunking is where an engineer meets the ceiling in practice, and the comment above a splitter that states how many characters a chunk may hold is the bet written down. What it is betting on is a ratio between characters and tokens, and that ratio is neither a constant nor an accident.

Why a subword vocabulary compresses English and not YAML#

The four-characters-per-token figure is not a myth, and it is worth knowing where it comes from before deciding how much weight it can carry. A subword tokenizer earns its compression ratio by holding whole common words in its vocabulary, so information costs one token, the costs one token, and ordinary English prose written for humans compresses to somewhere between four and five characters per token on a 30,000-entry WordPiece vocabulary, WordPiece being one of the standard schemes for choosing which word fragments earn an entry. That average was measured over the kind of text the vocabulary was fitted on, which is web pages, books and Wikipedia.

The causal step under that average is the one that predicts the next case. A vocabulary has a fixed number of entries, and those entries are spent on whatever was frequent in the text the vocabulary was fitted on. A string that was frequent there gets an entry of its own and costs one token however many characters long it is. A string that was not frequent there has no entry, so the tokenizer falls back to the longest fragments it does hold, and a run of underscores, hex digits and hyphens matches almost nothing longer than two or three characters. The compression ratio is therefore not a property of the tokenizer at all. It is a property of the overlap between your text and the text the vocabulary was fitted on, which is why text that does not resemble that distribution decomposes into two- and three-character fragments and the ratio collapses.

What one rule does to seven corpora#

Every row below is the same 2,000-character input measured through a real tokenizer, so the differences are entirely a property of the text.

Against a 512-token limit, three of those seven rows are already over, and the worst is over by a factor of two. Read the chars/token column as gaps rather than as levels, because the gaps say something the levels do not. The largest single step in the five bge rows is from a Markdown API reference at 3.8 to Python source with docstrings at 2.8, which is a bigger drop than the one from marketing prose to Markdown above it and bigger than either of the steps below it. The ratio does not decline smoothly as a document gets more technical. Markdown is prose with code in it and still tokenizes roughly like prose; Python source is code with prose in it. The collapse happens at that boundary, and the rows underneath are refinements of a text that has already stopped being made of words.

That shape matters for sizing. It means a corpus does not drift gently towards the danger zone as it accumulates technical material; it sits in the safe band until the composition of a document tips, and then it lands near the bottom of the column in one move. The JSON row is worth unpacking because it is the shape most internal corpora drift towards. A UUID such as 4f9d2c1a-8b7e-4c3d-9f10-2a6b5e8c7d31 is 36 characters and 21 tokens, because none of the hex runs are words, which works out at under two characters to the token and is roughly what a string containing no word at all costs. An ISO timestamp such as 2026-03-14T09:41:07.482Z is 24 characters and 14 tokens. A single line of a Kubernetes event — a pod name with a hash suffix, a namespace, a reason and a message — routinely costs one token per two characters. Code and configuration are dense in exactly the characters that a word-oriented vocabulary has no entries for: underscores, dots, brackets, hyphens, digits, and identifiers assembled out of parts.

The last two rows change the tokenizer as well as the text, and they are in the table for a reason the first five cannot show. A multilingual vocabulary has to spread its entries across many languages rather than concentrating them on one, so German prose lands at 3.3 where English marketing prose on the English-only vocabulary reached 4.6, and Japanese prose at 1.8 is the densest row in the table. Changing the model changes the exchange rate as surely as changing the document does.

The character rule is tempting for reasons that have nothing to do with laziness. len(text) is free and needs no dependency, whereas counting tokens means loading a tokenizer into the indexing process, which is a model download, a version pin, and a few hundred megabytes of memory in a job that may otherwise be pure text handling. Most off-the-shelf splitters default to characters as well, so the character rule is what you get by writing no configuration at all, and it produces chunks that look correct in every human inspection you will ever do of them. The rule also passes its own smoke test, because whoever wrote it checked a few articles by hand, and hand-checked articles are prose.

That last clause is the one to name carefully, because it is the shape most failed defences have. A characters-per-token divisor is a check whose output depends only on the length of the string it is handed. The quantity it exists to discriminate — how densely this particular document tokenizes — is precisely the quantity it never reads. Hand it 2,000 characters of Japanese prose and 2,000 characters of marketing copy and it returns the same estimate for both, while the tokenizer returns 1,124 for one and 431 for the other. A check that returns the same answer whether or not the fault is present is not a weak check; it is not a check at all. It feels like one because its answer happens to be correct on the documents anybody thinks to test it against.

The observed range across those seven samples is 431 to 1,046 tokens for a fixed 2,000-character input, which is a factor of 2.4. A margin of 12 tokens between an estimated 500 and a limit of 512 is not a margin against variance of that size; it is a margin against rounding. Put the two quantities side by side and they are not even the same kind of thing: the margin is a rounding error's worth of the limit, and what it would have to absorb is more than a doubling. The defence that the rule of thumb is "close enough" is a claim about the average, and the average is not what breaks. The tail of the distribution breaks, and the tail is systematically made of one kind of document.

Measuring the ratio on a corpus of your own#

If you want a defensible number for your own documents rather than a table from someone else's, the measurement takes a few minutes and produces a distribution rather than a single ratio:

import numpy as np

ratios = []
for doc in sample_documents(2000):          # a random sample, not the first 2000
    n_tokens = len(tok(doc, add_special_tokens=False)["input_ids"])
    ratios.append(len(doc) / n_tokens)

r = np.array(ratios)
print(f"p50 {np.percentile(r, 50):.2f}  "
      f"p05 {np.percentile(r, 5):.2f}  "
      f"p01 {np.percentile(r, 1):.2f} chars/token")
# p50 3.71  p05 2.24  p01 1.88 chars/token

The percentile that matters is the low one, because that is the document that produces the longest token sequence for a given character count. Sizing a character rule off the median gives you a chunker that is correct for half of that corpus; sizing it off the first percentile gives you a chunker that is correct everywhere and wastes about 60 percent of the available budget on ordinary prose.

That 60 percent looks wrong against a p50 of 3.71 and is right, and the reason is that it is not measured against the median. A chunker sized on p01 divides its character allowance by 1.88, which is to say it assumes every document it will ever see is as dense as the densest hundredth. Ordinary English prose is nothing like that dense: the marketing-prose row of the table above tokenizes at 4.6 characters per token. The p01 figure of 1.88 is a shade over two fifths of 4.6, so a prose document sized under the p01 rule occupies a shade over two fifths of the positions available to it and leaves close to three fifths of them empty. The waste is measured against the text the rule is over-cautious about, and not against the middle of the distribution where the rule is roughly right. Neither setting is a good trade, which is the argument for splitting on token counts directly rather than hunting for a safer character constant.

Reading the real limit off the model#

The number that governs truncation is a property of the loaded model object, and it is not always the number printed in the model card. Three related values exist and they can disagree.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-base-en-v1.5")

print(model.max_seq_length)                                # 512
print(model.tokenizer.model_max_length)                    # 512
print(model[0].auto_model.config.max_position_embeddings)  # 512

They can disagree because they are answers to three different questions. max_position_embeddings is the architectural ceiling — the number of position vectors the encoder actually has, which is the count derived above. model.max_seq_length is the value the sentence-transformers wrapper passes to the tokenizer as max_length when it truncates, and it is loaded from sentence_bert_config.json in the checkpoint, which is the bundle of weight and configuration files that a published model consists of. model_max_length is a third record, written into the tokenizer's own configuration, and it can be edited independently of the other two. Agreement between the three is a convention rather than something enforced at load time.

The architectural ceiling and the wrapper's setting are equal for most encoders and are not equal for some: a checkpoint whose backbone supports 8,192 positions can ship with max_seq_length set to 512 because that is what it was fine-tuned at, and the wrapper will happily clip at 512 while the model card advertises 8,192. It is also assignable, so model.max_seq_length = 256 is a one-line change that silently shortens every chunk in your index. The sentence-transformers documentation on computing embeddings states the behaviour plainly — input longer than max_seq_length is truncated, without a warning — and that page is the right thing to link in a code review when someone proposes a character-based splitter.

The budget is smaller than the limit#

The limit counts everything in the sequence, and your text is only part of the sequence. Three other things take from the same 512.

Special tokens come first. A BERT-family encoder wraps the input in [CLS] and [SEP], and an XLM-R-family encoder wraps it in <s> and </s>; either way, two positions are gone before any content is encoded.

tok = model.tokenizer
text = "kubectl rollout restart deploy/api -n prod"

print(len(tok(text)["input_ids"]))                            # 15
print(len(tok(text, add_special_tokens=False)["input_ids"]))  # 13
print(tok.num_special_tokens_to_add(pair=False))              # 2

Two readings in that block agree, which is more useful than either alone. The tokenizer reports its own answer directly, and the same figure falls out of subtracting the encoding without special tokens from the encoding with them: fifteen against thirteen leaves two. The overhead is therefore a real property of the encoding rather than an artefact of how one of the counts was taken, and it is a constant, so it does not need to be re-measured per document.

Then there is the instruction prefix. Topic 8.6 established that retrieval encoders expect a different prefix on the query side and the document side, and that prefix is part of the encoded sequence, so it is spent out of the same budget on every chunk you ever index. "passage: " costs 3 tokens on E5's tokenizer. The longer instruction-style prefixes that BGE and Instructor-family models accept — something along the lines of Represent this document for retrieval: — cost 8 to 20 tokens depending on how specific you made the task string.

Prepended metadata is the largest and the most often forgotten, because it is usually added by a different function than the one that does the chunking. A common header looks like this:

Source: runbooks/payments/reconciliation-failures.md
Section: Recovering from a stuck settlement batch
Updated: 2026-03-14

That is 133 characters and 41 tokens through the bge tokenizer, mostly because the path fragments and the date shatter into pieces. Together with a 3-token prefix and 2 special tokens, 46 of the 512 positions are committed before a single word of the article is encoded, which leaves 466.

The itemisation is worth checking rather than reading past. Forty-one for the header, three for the "passage: " prefix and two for the special tokens comes to forty-six, and forty-six taken from 512 leaves 466. The sum closes with nothing left over, and what that licenses is the strong form of the claim: every position unavailable to the article's own text has been named. Were some other part of the assembly quietly consuming positions — a separator between header and body, a trailing newline the template carries — the itemised total would fall short of the measured overhead, and the shortfall would show up as a residual with nothing to attribute it to. There is no residual, so the budget is 466 rather than approximately 466.

A chunker that targets 500 tokens because the limit is 512 is over budget on every chunk it produces, uniformly, and the amount it loses is exactly the length of the header — which is to say, it loses the last 34 tokens of every article and keeps the filename. That 34 is the same identity read the other way round. A chunker aiming at 500 against a budget of 466 overruns by the difference between the two, and the difference between 500 and 466 is 34. Neither number depends on the document: the target is a constant somebody configured, and the budget is a constant of the model plus the template. So the loss is not a tail risk that only appears on long articles. It is a fixed quantity removed from the end of every chunk in the index, and because it comes off the end, what survives is the header and the opening — which is the part a human reviewer reads first when spot-checking.

Computing that budget once at startup, from the objects themselves, removes the whole class of error:

def effective_budget(model, prefix: str, header_template: str) -> int:
    """Tokens available for chunk body text, given everything else we prepend."""
    tok = model.tokenizer
    special = tok.num_special_tokens_to_add(pair=False)
    overhead = len(
        tok(prefix + header_template, add_special_tokens=False)["input_ids"]
    )
    return model.max_seq_length - special - overhead


BUDGET = effective_budget(model, "passage: ", HEADER_TEMPLATE)  # 466
assert BUDGET > 0, "prefix and metadata consume the entire sequence"

Headers vary in length across documents, so what the template gives you is a floor on the overhead rather than an exact figure — a document whose source path or section title runs longer than the template's spends more than 41 tokens on its header, and the budget it actually receives is correspondingly smaller than 466. The honest version measures the real assembled string per chunk rather than a template. That is a per-chunk measurement, which is the subject of chapter 6 (where per-chunk token counts and retained ratios are turned into something a monitor can alert on).

Tokens, bytes, or characters — the unit is not always tokens#

When the encoder runs in your own process, the unit is unambiguously tokens of a tokenizer you can call. When it runs behind an HTTP API, the documented limit may be counted in a unit you cannot reproduce locally, and the three units fail differently.

The three rows bind at different places on the same document, which is why the unit has to be recorded rather than assumed. A Terraform file is heavy in tokens and cheap in bytes, since every character in it is one byte of ASCII, so a byte cap is unlikely to be what stops it. A Japanese article is heavy in both, at 3 bytes to the character on one side and 1.8 characters to the token on the other, so it can be rejected by a limit it passed comfortably when the same document was measured in a different unit. Nothing about a document tells you in advance which of the three will bind, which is why a number without its unit attached is not a limit anybody can plan against.

The unit can also change under you without the number changing. A model family that ships a v2 checkpoint with a rebuilt vocabulary keeps the same advertised 512 and quietly changes how much of your text fits inside it, because a vocabulary tuned on more code will compress code better and shift every chunk's token count by a few percent. That is a small effect compared with the differences between text types above, and it is large enough to move a corpus that sat at 505 tokens across the line.

There is also a distinction between the per-item limit and the per-request limit, and only one of them is usually in the model card. A batch endpoint may accept 512 tokens per input and simultaneously reject a request whose total payload exceeds a few megabytes or whose array exceeds a few hundred items, and the second failure at least tends to be loud. Recording the unit alongside the number, in the same config your indexing pipeline reads, is what makes the value auditable later; a constant named MAX_TOKENS = 512 with no note of whose tokens is a value nobody can check.

The limit is not the chunk size, and it is not the context window#

Knowing the limit is not the same as knowing the right chunk size. A model that accepts 512 tokens may still retrieve better at 250, for reasons that have nothing to do with truncation and are the subject of chapter 4 (where a single pooled vector is shown to get blurrier as the passage it summarises gets long). 512 is a ceiling on what may be embedded at all, and the best chunk size is a separate, empirical question living underneath that ceiling.

The encoder's maximum sequence length is also unrelated to the context window of whatever generation model consumes the retrieved passages later; a 200,000-token context downstream does not extend the encoder by a single position, and the two numbers appearing in the same architecture diagram is the most common source of the confusion. The two limits belong to two different models, with two different tokenizers, applied at two different times. The encoder's ceiling binds when a passage is indexed, long before anybody asks a question, and the generator's binds on the prompt assembled at answer time. A token counted under one of them is not a token counted under the other, and neither number constrains the other in any direction.

Everything above is a statement about your chunker and the model's documentation. The limit is counted in the model's tokens, and a character count is a proxy for it whose accuracy varies by a factor of 2.4 across the seven samples above and shifts again whenever a vocabulary is rebuilt underneath you. The budget your text actually receives is the limit minus the special tokens, minus the prefix, minus whatever metadata another function prepends, which for the header shown here is 466 rather than 512. Both facts are measurable in a few minutes from objects you already have loaded, and neither is visible from the outside once the vectors are written.

None of it says what the code between your chunker and the model actually does. You can count tokens perfectly, size chunks to 466, and still be shipping clipped vectors because a wrapper three layers down sets its own max_length, or because the provider enforces a cap lower than the one it publishes. Counting proves your chunker matches the docs, and the failure this topic is about happens in the space between the docs and the running system. The next chapter is a five-minute experiment that closes that space.