Token counts and retained ratios make an invisible defect alertable

An indexing pipeline — the batch job that reads source documents, splits them into chunks, embeds each chunk and writes the resulting vectors into a search index — can lose the back half of every chunk it processes and still report a completely ordinary run. Truncation, meaning the encoder discarding whatever tokens sit past its maximum input length, is documented behaviour rather than an error condition, so nothing is raised. Duration does not move, because clipping the input makes the encoder's work smaller rather than slower. Chunk count does not move, because splitting happens upstream of the encoder and is untouched by what the encoder later does with what it was handed. Vector count does not move, because a clipped input still produces exactly one vector, and error rate does not move because no error occurred.

Every one of those numbers is insensitive to the only quantity that changed, which is how much of each chunk actually reached the model. Four fields per chunk supply a number that is sensitive to it. They are cheap enough to compute unconditionally, they add under a percent to the storage a vector already occupies, and they convert a defect that otherwise surfaces as a vague complaint weeks later into a job that fails before its index is promoted.

The four fields#

For every chunk, at the moment it is about to be embedded, record its token count, the effective limit in force, whether truncation occurred, and the retained-token ratio. The first three are almost free once the tokenizer is already loaded, and the fourth is derived.

@dataclass(frozen=True)
class ChunkTokenStats:
    token_count: int          # tokens in the assembled input, specials included
    effective_limit: int      # measured, not read from the model card
    truncated: bool
    retained_ratio: float     # min(token_count, effective_limit) / token_count


def measure(assembled_text: str, tok, effective_limit: int) -> ChunkTokenStats:
    n = len(tok(assembled_text, add_special_tokens=True)["input_ids"])
    return ChunkTokenStats(
        token_count=n,
        effective_limit=effective_limit,
        truncated=n > effective_limit,
        retained_ratio=min(n, effective_limit) / n,
    )

The ratio is worth reading out of that expression rather than taking on trust, because its behaviour splits cleanly in two and every later table depends on the split. When a chunk is at or under the limit, the smaller of the two quantities is the token count itself, so the fraction is the token count over the token count, which is exactly one for every such chunk regardless of its length. When a chunk is over the limit, the smaller quantity is the limit, so the fraction becomes the limit divided by the token count. That second form falls as the chunk gets longer, since the numerator is fixed by the deployment and only the denominator grows. A retained ratio is therefore a flat 1.0 across the whole healthy population and a strictly decreasing function of length across the truncated one.

The boolean and the ratio are both needed because they answer different questions. The boolean counts incidents, which is what an alert threshold is set on. The ratio measures severity, which is what tells you whether to stop the world: a corpus where every truncated chunk sits at 515 tokens against a 512 limit is losing three tokens of trailing punctuation and can wait until Monday, while a corpus at a mean retained ratio of 0.61 has lost two fifths of its text and should not be promoted. Reporting only the boolean makes those two look identical, and reporting only the ratio makes a corpus with one catastrophically long chunk and 99,999 clean ones look fine on the average.

effective_limit earns its place by being a recorded value rather than a constant in the code. It is the number produced by the boundary sweep (chapter 2, where an input is grown a little at a time until the vector stops changing, which gives the length at which the deployed system actually cuts) measured against the deployed path, stored in the same configuration the indexing job reads and refreshed when that sweep re-runs.

Hard-coding 512 there because that is the figure on the model card is the mistake worth naming carefully, because it is a defence that returns the same answer whether or not the fault it exists to catch is present. A container clipping at 512 and a container clipping at 256 both produce chunks whose token counts sit below the hard-coded value, so n > effective_limit is false in both cases and every chunk is written out with truncated=False, retained_ratio=1.0. The field varies, the fault varies, and the two never vary together. A check whose output is constant across the thing it is meant to discriminate carries no information about that thing, however carefully it is computed.

The belief is tempting for a reason worth stating, since it is not carelessness. A model card is authoritative, versioned, and written by the people who trained the model, and the number on it is genuinely true — 512 really is that model's maximum sequence length. What it is not is a fact about the container that is running, and the gap between those two is precisely the gap this instrumentation exists to measure. Instrumentation calibrated against documentation reports the health of the documentation.

Where in the indexing path the measurement belongs#

The measurement has to happen after everything that will be sent has been assembled and before the call that sends it, and that placement follows from three checkable steps rather than from taste, ruling out two tempting locations along the way. The first step is that the encoder receives exactly one string per chunk — the assembled input, meaning the task prefix, the metadata header and the chunk body concatenated — and the clip is applied to that string and to nothing else. So the only length that predicts what will be lost is the length of the assembled input, and any measurement of some other string is measuring something the model never sees.

The second step rules out the chunker. A chunker runs before assembly, so the longest string available to it is the chunk body alone, missing the prefix and the header. It therefore measures short by exactly the length of what has not been prepended yet, which for the token budget of chapter 1 — where the 512 positions are counted down by a 3-token prefix, a metadata header and 2 special tokens, leaving 466 for the article itself — is 46 tokens. A chunker reports 466-token chunks as comfortably under a 512 limit while the assembled input is 512 exactly, and both statements are true about different strings.

The third step rules out the model wrapper, for a different reason and with a different remedy. A wrapper that batches pads every member of a batch out to a common width so the tensor is rectangular, which means the length visible inside it is the width of the longest member rather than the chunk's own. The wrapper also frequently belongs to somebody else, so instrumenting it is a fork rather than a patch. Between assembly and the send is the only position where the string being measured and the string being encoded are the same string, which is why the measurement sits there.

Both outputs are worth having, and they serve different people. The aggregated metrics — a counter of truncation events and a histogram of retained ratios, both labelled by corpus, source system and model version — are what a dashboard renders and what an alert rule evaluates. The per-chunk fields stored alongside the vector are what someone queries at 09:00 the next morning to answer "which documents were affected", and without them the answer requires re-running the whole tokenization pass over every document to reconstruct what the job already knew.

A typical emission looks like this, one structured line for the run and metrics for the series:

{
  "event": "index_run_complete",
  "run_id": "idx-2026-03-19-0200",
  "corpus": "engineering-docs",
  "model": "bge-base-en-v1.5",
  "effective_limit": 510,
  "limit_measured_at": "2026-03-17T04:11:00Z",
  "chunks_total": 58913,
  "chunks_truncated": 4,
  "truncated_share": 0.000068,
  "retained_ratio_p50": 1.0,
  "retained_ratio_p05": 1.0,
  "retained_ratio_min": 0.62,
  "worst_chunk_id": "runbooks/payments/reconciliation-failures.md#3"
}

Two of those values are derived from the two above them, and checking that they close is the fastest way to tell a real emission from a plausible-looking one. Dividing the 4 truncated chunks by the 58,913 total gives the 0.000068 share, and that same fraction expressed per hundred rather than per one is the 0.007 percent this run would appear as on a dashboard. The p05 ratio of 1.0 sitting next to a minimum of 0.62 is the other consistency check: four chunks out of 58,913 is far fewer than one in twenty, so the fifth percentile of the ratio distribution is still drawn from the untruncated population, where the ratio is one by construction. A p05 below 1.0 on a run with a handful of truncations would mean the two figures were computed over different sets of chunks.

worst_chunk_id is the field that gets used most and is the one most often left out. An aggregate tells you something is wrong and a document identifier tells you what to look at, and the marginal cost of tracking a running minimum is one comparison per chunk.

The cost of all this is small enough not to need a justification meeting, and it is worth having the figure to hand anyway. Tokenizing is already happening if the chunker is token-aware, so measure() reuses that work; where it does not, tokenizing 59,000 chunks with a fast Rust-backed tokenizer takes around 12 seconds on a single core, against an embedding step of 34 minutes on a GPU. The storage figure follows from the vector's own geometry. A float32 component is four bytes wide, and a vector of 768 dimensions carries one component per dimension, so the vector is 768 multiplied by four, which is the 3,072 bytes it occupies on disk. Four extra fields per chunk add roughly 24 bytes against that, and 24 against 3,072 is well under one percent. The reason instrumentation like this is missing from most indexing jobs is not its cost; it is that nobody writes a metric for a failure mode they have not yet had.

A metric measured after the truncation reports perfect health forever#

Two mistakes in the measurement produce a dashboard that stays green for the life of the job, and both look like the careful thing to do.

The first is calling the tokenizer with the same arguments the encoder uses. A job that assembles its input with truncation=True, max_length=510 and then takes len(input_ids) is counting the identifiers that survived the clip rather than the ones that arrived, because the clip happens inside the tokenizer call and before anything is handed back. The count is therefore capped at 510 by construction. Every downstream field follows from that cap: n > effective_limit compares a number that cannot exceed 510 against 510 and is never true, and min(n, effective_limit) always selects n, so retained_ratio is the token count over itself, which is 1.0 on every chunk of every run.

That is the hard-coded-limit defect one layer further in, and it is worth seeing as the same shape. There the recorded limit was constant while the deployed one varied; here the measured length is clipped by the very mechanism it exists to detect, so the instrument can only ever report health. Reusing the exact call the encoder makes is normally the right instinct, and that is what lets this survive review — measuring with different arguments is the usual way to end up with a count that is true of a string the model never received. The instinct inverts on exactly one argument, which is the one that changes the string rather than merely describing it.

The tell is the shape of the distribution rather than any single value: token counts pile up on one integer, so a corpus spreading smoothly from 40 to 500 tokens starts reporting tens of thousands of chunks at exactly 510 and none at 511, and real text does not agree on a length. A canary at startup — a tiny check that runs on every boot and fails loudly when its one assumption stops holding — is cheaper than watching histograms for that shape.

probe = " ".join(["reconciliation"] * 2000)
n = len(tok(probe, add_special_tokens=True, truncation=False)["input_ids"])
assert n > effective_limit, "measurement tokenizer is clipping; counts are capped"

The second is counting with a tokenizer that is not the one doing the embedding. Reaching for cl100k_base because tiktoken is already imported for the generation side, against an index built on bge's 30,000-entry WordPiece vocabulary, gives numbers wrong in the direction that hides the problem: cl100k is a 100,000-entry byte-pair encoding vocabulary, built by repeatedly merging the most frequent adjacent pairs of symbols, and it was fitted on a mixture heavy in code, so it compresses YAML, paths and identifiers far better than a word-oriented English vocabulary. A runbook chunk costing 412 cl100k tokens costs 680 bge tokens, so an operational corpus reports comfortably under budget while a third of it is clipped. The milder version is loading the right tokenizer by name rather than off the model object, which drifts as soon as the served checkpoint moves to a revision the indexing code does not know about. Taking tok = model.tokenizer from the object that will encode removes both.

What the distribution looks like when it is fine, and when it is not#

A nightly run over the 6,200-document corpus of engineering documentation used for the chunk-size sweep (chapter 5, where one corpus is indexed at several chunk lengths and the arms compared against a fixed query set), chunked at a 384-token target against a measured limit of 510, gives the left-hand column. The middle column is the same 58,913 chunks on a night when a container's default input length was 256 rather than the length it had been configured for. The right-hand column is that same night as reported by a job making the first mistake above. The rows are quoted as percentiles, where p50 is the median and p95 is the value that 95 percent of chunks fall below, because a mean over 58,913 chunks hides exactly the tail that this instrumentation exists to find.

The healthy column has one row that is not an independent observation. Its retained-ratio minimum belongs to the longest chunk in the run, and the longest chunk is the 823 in the row three above it, so the minimum is the measured limit of 510 divided by 823, which is the 0.62 printed. Two instruments that never consult each other — a boundary sweep run against the deployed encoder, and a pass over the token counts of a corpus — land on the same figure. That agreement is the licence for reading the ratio rows as a restatement of the token-count rows rather than as separate evidence, and it is a cheap check to run on any real emission.

The middle column is the same identity applied three times, and it closes exactly. Dividing the dropped limit of 256 by the median token count of 341 gives the 0.75 median retained ratio. Dividing 256 by the maximum of 823 gives the 0.31 minimum. The third division is the one that looks misfiled: 256 divided by the 95th-percentile token count of 402 gives the 0.64 in the row labelled p05, not a row labelled p95.

The percentile flips end for end because the ratio, as the code above showed, is a decreasing function of length once a chunk is over the limit. Longer chunks are worse-retained chunks. Sorting chunks by length puts the longest at the top, and sorting the same chunks by retained ratio puts those identical chunks at the bottom, so a chunk sitting at the 95th percentile of length sits at the 5th percentile of retention. What that flip buys is a single reading of the whole column: three rows, three divisions, one division each, no residual and no term left over. Every unit of loss in the middle column is accounted for by one cause, which is the limit. Ratio rows that failed to reproduce from the token-count rows would mean some chunks were losing text for a second reason, and there is nothing left for a second reason to explain.

The threshold arithmetic follows from the four healthy truncations, which are an extraction fault rather than a chunker fault: three are pages of a generated API schema emitted without whitespace, and the fourth is a minified vendor bundle. Half of one percent of 58,913 chunks is a fraction over two hundred and ninety-four and a half, and a count of chunks is a whole number, so the boundary sits at the 294 quoted. Four chunks is far below it and produces a warning carrying four document ids, while 51,204 chunks is far above it and fails the job outright. The share that number produces is 51,204 divided by 58,913, which is the 86.9% in the row above, against the 0.007% the same division of 4 by 58,913 produces in the healthy column.

The right-hand column is worth recognising on sight, because its shape diagnoses itself. A measurement that clips before it counts maps every value above the cut onto the cut, since a count of what survived can never exceed the length survival was capped at. Everything longer than 256 is therefore reported as 256, the distribution acquires a spike at that integer and no mass at all above it, and the median, the 95th percentile and the maximum have nowhere to go but onto the same value. Real text does not agree on a length, so a corpus whose p50, p95 and maximum are one integer is making a statement about the instrument rather than about the documents.

The ratio rows carry what the share cannot, and the case that shows it is one where the share is useless. A chunker targeting 500 body tokens against the 466-token budget from chapter 1 truncates nearly every chunk, so its share reads close to 1.0 and is indistinguishable from the middle column. Its retained ratios cluster around 0.94 rather than the 0.75 of the middle column, and that figure takes one step to see. The chunker's 500 tokens are body only, and by the time the header, the prefix and the special tokens have been prepended, the assembled input is 500 plus those 46 tokens, which is a little over five hundred and forty tokens. Dividing the measured limit of 510 by that assembled length gives a number a touch under the 0.94 quoted, which is what "around 0.94" describes at the precision the row offers. Dividing the budget by the target instead gives a much lower figure and is the wrong comparison, because the limit applies to the assembled string. Both cases should stop a release, and they route to different people: one is an arithmetic error in the budget, and the other is a deployment that changed underneath the indexing job.

Which numbers deserve which response#

An indexing job that sizes chunks by token count against a measured limit should truncate nothing, so the healthy value is zero and any non-zero value is a defect rather than a degradation. That argues for a strict threshold, and strict thresholds on a corpus that ingests arbitrary user content produce alert fatigue, because one pathological document — a 40,000-token minified bundle that slipped past extraction — will trip it. The resolution is to separate the two failure shapes by scale.

The distinction between the first two rows is what makes the strict threshold survivable. A handful of individually pathological documents produces a small share and a warning with a list of ids, which someone fixes in the extraction rules. A configuration change or a model swap produces a large share within one run, which fails the job before the index is promoted. The retained_ratio_p05 row catches the case the share alone misses, where a chunker targeting 500 tokens against a 466-token budget clips every chunk by a little and the share is 100 percent but the damage per chunk is small — that combination should still stop the release, because it means the budget calculation is wrong. Setting that row's threshold just under 1.00 rather than at some lower value works because a healthy population reports exactly 1.00 at every percentile, so any movement at all is a signal.

The query-side row exists because query truncation is real and asymmetric in its consequences. Most queries are ten to thirty tokens and could not truncate if they tried, and a small minority are a pasted stack trace or an entire error log, which are exactly the cases where a user has the most specific information and the greatest need. Those queries get clipped to their first 510 tokens, which for a stack trace means the exception header and none of the frames. Logging the query hash rather than the query text keeps that instrumented without putting user input in the metrics pipeline.

The nightly sweep row is the one that would have caught the Tuesday edit above, and it is the only row that would. Every other signal in the table is computed against the recorded limit, so a change to the deployed limit moves none of them until the recorded value is corrected. Re-measuring the boundary on a schedule and alerting on any movement is what keeps the rest of the instrumentation honest, and it is the reason the boundary sweep belongs in a cron entry rather than in a one-off investigation.

A green truncation dashboard is not evidence of good chunking#

These four fields measure input, not quality. A run reporting zero truncation and a retained ratio of 1.0 across every chunk is reporting that every token reached the encoder, and that is entirely compatible with retrieval being poor for the reasons dilution sets out (chapter 4, where one pooled vector standing for a long passage that spans several subjects matches each of them mediocrely), since dilution has no per-chunk observable and will not appear here or anywhere near here. Treating a green truncation dashboard as evidence that chunking is correct is the misreading to guard against, and it is an easy one to fall into precisely because these fields are the only numbers in the vicinity.

They also do not validate the chunk boundaries, the extraction, or the metadata. A retained ratio of 1.0 on a chunk consisting of a navigation menu and a cookie banner is a truthful report about a worthless chunk. The fields answer one question completely and no others at all, which is the property that makes them safe to alert on and dangerous to reason from.

What they do is close the loop that the rest of this topic opened. Counting tokens with the model's own tokenizer establishes the budget; the boundary sweep establishes what the deployed system actually enforces; the tail test (chapter 2, where a distinctive tail is appended to a long text and the two vectors compared for a difference) and the chunk-size sweep establish what the loss costs; and these four fields make all of it continuous. The token count and the recorded limit are the two quantities the whole topic has been about, and the boolean and the ratio are what happens when you subtract one from the other on every chunk instead of once during an investigation.

That continuity is the whole return. The next time a manifest changes or a corpus drifts towards code, the job fails at 02:38 with a document id attached, instead of surfacing as a vague complaint eleven days later that sends someone to the reranker. From here the work is applying it — measuring your own limit, proving your own client's behaviour, and putting the four fields into the indexing path you already have.