A client that accepts an over-long chunk has already discarded text

Someone on the team pastes a 4,000-token chunk through the embedding client — the code path that turns a string into a vector — as a sanity check, expecting a stack trace. What comes back is a 768-dimensional float array, in 90 milliseconds, with no warning on stderr and nothing in the logs. The natural reading of that result is that the model handled it, so perhaps the limit is higher than the model card says, or perhaps the client is doing something clever. The correct reading is that roughly seven eighths of the text was thrown away somewhere between the function call and the encoder, and the vector you are holding describes the first 510 tokens only.

Four terms carry the rest of this. An embedding model reads text as tokens, the pieces its tokenizer cuts the input into — mostly whole common words, with fragments for anything its vocabulary does not hold. Every such model has a maximum sequence length, the largest number of tokens it can encode in one call, fixed because the encoder was built with a set number of position slots and there is no slot past the last one. Truncation is what almost every client does when that ceiling is crossed: keep the leading tokens, drop the rest, return an ordinary vector, record nothing anywhere. The quantity worth measuring is the effective limit — the token count at which your own running deployment actually begins dropping text — which is a property of the assembled system rather than of the model, and is frequently not the number printed on the model card.

An ordinary vector for an eighth of the input#

The seven eighths comes out of a division worth doing rather than accepting. Of the 4,000 tokens that went in, 510 came out the other side described by the vector. Multiply those 510 surviving tokens by eight and the product overshoots the 4,000 that were pasted, so the survivors are a fraction slightly larger than one part in eight. Everything else went nowhere. Seven of the eight parts, near enough, are the discarded share, which is where "roughly seven eighths" comes from.

Silence is the default because truncation is the sane library default. A splitter that raises on long input turns every ingestion job into a crash loop the first time someone commits a large file, so the ecosystem settled on clipping. The Hugging Face documentation on padding and truncation lays out the strategies and their defaults, and the important detail is that truncation is a per-call argument with different effective values depending on who calls it: raw transformers tokenizers do not truncate unless asked, while sentence-transformers passes truncation=True, max_length=self.max_seq_length on every encode. So the same model, reached through two different code paths, gives you a hard error in one and a quiet clip in the other.

That difference is the reason the observation at the top is not evidence of anything. Clipping is not an error condition inside a library that was designed to clip; it is the documented behaviour of the argument, executed successfully. Nothing raises, so no handler runs. Nothing is anomalous, so no warning is emitted. And the return type is the same float array either way, so the call site has no field to inspect even if it wanted to.

Four behaviours behind one ordinary vector#

The observable outcome — a normal vector, no error — is consistent with several different mechanisms, and they have different fixes.

Read the middle column rather than the rows. Only the first row produces an observation different from the others, and it is the only one that is safe. The second and third rows are indistinguishable from each other at the call site, because both return a normal vector built from a shortened input, and they differ solely in which machine did the shortening. That distinction matters for the fix and not for the symptom.

The fourth row is the one that breaks naive testing, and it is common enough in wrapper libraries and vendor cookbooks to be worth ruling out explicitly. A helper that quietly splits a 4,000-token input into eight windows, embeds each, and returns the average produces a vector that responds to every part of the input, so no test based on "did the vector change" will flag it. It is also not obviously wrong — an averaged vector is a defensible thing to build — but it is a different object from what the rest of your index contains, and nothing in the return type tells you which one you got.

Four maximum lengths on one call path#

The mechanism that makes silent loss possible is that the limit is not one number. A concrete path makes the layering visible. An indexing job calls a framework's embed_documents, which calls a provider SDK client, which posts to a self-hosted text-embeddings-inference container, which loads the checkpoint. There are four places to set a maximum length: the framework wrapper's constructor argument, the SDK's per-request field, the container's --max-client-batch-size and auto-truncate flags, and sentence_bert_config.json inside the checkpoint. Whichever is smallest wins, and none of the four is visible from the call site.

The reason the smallest wins is worth stating as steps, because it is what makes the failure irreversible rather than merely confusing. Each layer applies its own limit to whatever arrived, by dropping the tokens past that limit. It then hands on what is left, and what is left is all it hands on — no layer forwards a copy of what it removed, and no layer annotates the request to say that it removed anything. So a layer further down the path never sees the tokens an earlier layer took out. Having never seen them, it cannot encode them, whatever its own limit happens to be. A generous limit downstream is therefore inert: it can only fail to clip text that is already gone. The binding constraint is the minimum over the four, and the ordering of the layers changes which config file you edit, not what the effective limit turns out to be.

One of the four is a requirement and three are decisions, and separating them tells you which arguments are worth having. The count of position slots inside the encoder is a requirement: the weights were trained with that many, and there is no slot to add a token to. The value in sentence_bert_config.json is not that number. It is a decision recorded by whoever published the checkpoint, usually the length the model was fine-tuned at, and it is an ordinary assignable attribute at runtime. The remaining three — the wrapper's constructor argument, the SDK field and the container flag — are decisions made by whoever deployed the thing, often by someone who never read the model card at all.

The token count that never makes a call#

Reading the documentation separates none of these. The model card documents the model, your client library documents its own defaults, and the provider documents its API, but the thing you actually want to know is a property of the composition of all three plus whatever wrapper your framework inserted. Counting tokens with the model's tokenizer and confirming every chunk is under 512 is a check on your chunker; it is silent about all four rows of that table, because it never makes a call.

That check is a genuinely good thing to have, which is why it is such a comfortable place to stop. It is fast, it runs offline, it produces a clean pass, and it catches the mistake from chapter 1 exactly — a chunking rule written in characters that overshoots when the same text is measured in tokens. Nothing below is an argument for deleting it.

The trouble is what its answer depends on. The inputs to a token count are the chunk text and the tokenizer, and nothing else. The length at which the deployed container clips is not one of those inputs, and a function cannot vary with something it never reads. So the check returns the same clean pass whether the running container clips at 512 or at 256 — the same answer in the presence of the fault and in its absence. A defence that is constant while the thing it must discriminate varies is not a weak defence; it is not a defence against that fault at all, and its reliability against the fault it does catch is exactly what makes the mistake tempting.

Put concretely: a chunking job where every chunk is verified at 480 tokens and the deployed container clips at 256 will pass the token check on every chunk, every run, forever, while half of each chunk is discarded. What the tail test below adds is the missing input. It calls the deployed path, so the deployed limit is one of the things determining its output, and its answer therefore changes when the deployment changes. The count validates the input against a belief about the system; the test validates the belief.

The tail test#

The experiment that settles it takes about five minutes and rests on one fact: an encoder is deterministic in its input, so if any token of the input changes, the output vector changes. Embed a long text, embed the same text with a distinctive tail appended, and compare the two vectors. If they are identical, the tail was never encoded.

Two conditions sit under that fact, and both matter for reading the result. Determinism holds for a fixed model on fixed hardware at a fixed numeric precision, and the test measures rather than assumes it. And a changed token moves the vector because every output dimension is pooled from every input position, so a position that is present contributes to all of them, while a position that was dropped before the forward pass cannot contribute to any. That is why the probe belongs at the end of the text: truncation keeps the leading tokens, so a probe at the front would survive every clip and tell you nothing.

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-base-en-v1.5")

body = open("long_runbook.md").read()          # ~3,900 tokens
tail = " The escalation contact is the platform on-call rota, code PLTFRM-7."

v_control_a = model.encode(body)
v_control_b = model.encode(body)                # same input, twice
v_with_tail = model.encode(body + tail)

noise  = np.abs(v_control_a - v_control_b).max()
signal = np.abs(v_control_a - v_with_tail).max()

print(f"noise floor      {noise:.3e}")   # 0.000e+00 on CPU fp32
print(f"tail difference  {signal:.3e}")  # 0.000e+00  -> tail never reached the model

The control pair is not decoration. On GPU with mixed precision, meaning parts of the forward pass are computed in 16-bit floats for speed, or with a batching layer that pads to different lengths on different calls, embedding the same string twice can differ by around 1e-7 per element. A test that compares against exact zero will then report a difference for two identical inputs, and you will conclude, wrongly, that nothing is being truncated. Measuring the noise floor first gives you a threshold with a basis: if the tail difference is the same order of magnitude as the control difference, the tail changed nothing.

The tail itself has to be chosen with a little care. It should be short enough that its own token count does not matter, and lexically distinctive enough that it would move the vector measurably if it were encoded — a proper noun, an identifier, a code. Appending a comma or a newline is a weak probe, because some tokenizers normalise trailing whitespace away and you will get a false positive for truncation from a preprocessing step rather than from a length limit.

Finding the cut point by binary search#

A negative result on the tail test is informative and a positive result is only half an answer, because it tells you text is being lost without telling you where. Growing the input and finding the length at which the vector stops responding gives you the cut point in your own tokens, and that number is comparable against the documented limit.

def effective_limit(model, filler_tokens, tok, lo=8, hi=8192) -> int:
    """Smallest n where appending a tail after n tokens no longer changes the vector."""
    probe = " QZX-4417 escalation code."

    def tail_matters(n: int) -> bool:
        head = tok.decode(filler_tokens[:n])
        a, b = model.encode([head, head + probe])
        return np.abs(a - b).max() > 1e-6

    while lo < hi:                              # binary search on the cut point
        mid = (lo + hi) // 2
        if tail_matters(mid):
            lo = mid + 1
        else:
            hi = mid
    return lo

# effective limit: 510 tokens (documented 512)

Halving is the right procedure here for a reason specific to this property. tail_matters is monotone in n: below the cut point every prefix leaves room for the probe, so the answer is true, and at or above it the probe is dropped, so the answer is false. A property that switches once and never switches back has a single crossing point, and each comparison against the midpoint tells you which half of the remaining range contains it. If the property could flip more than once, halving would be unsound and you would have to walk the range. Stating that assumption is also a way of noticing when it is violated: a split-and-average wrapper never returns false, and the search runs to the top of its range.

The threshold in the code is the noise floor argument turned into a constant. The comparison is against 1e-6, an order of magnitude above the roughly 1e-7 per-element wobble that mixed precision produces, which is what makes a fixed constant workable — provided you have measured your own noise floor first and found it below that line.

Filler matters here too: it must be text the model will not deduplicate or normalise, so a repeated sentence is a poor choice and a long stretch of real prose from your own corpus is a good one. Running the sweep with add_special_tokens=False counts on the tokenizer's side while the model counts with them, which is exactly why 510 rather than 512 comes back — the two special tokens from chapter 1, one at each end of the sequence, now visible as a measurement rather than as an argument.

That subtraction is worth doing explicitly, because what it licenses is the whole value of the next comparison. Take the documented 512, remove the two positions the encoder spends on its own boundary markers, and 510 positions remain for content. The measured cut point is 510. Two independent things — a documented number and a number produced by encoding real text through the running system — land on the same value once one known, nameable term is accounted for, and there is no residual left over. Nothing unexplained is being lost. That is what makes a cut point lower than 510 readable as a second cause rather than as slop in the instrument: the instrument has already demonstrated that it can account for its own discrepancies down to the last two positions.

The value of the cut point is what it lets you conclude when you hold it against the documentation.

The diagram is a decision procedure, and reading it as a chain rather than as a picture is what makes it usable. Embed the body and the body plus tail, then compare against the noise floor. A distinctive tail, if it were encoded, would move the vector well above that floor, which is the entire reason for choosing a distinctive one — so no movement means it was not encoded, and text is being discarded. If the vectors do differ, the tail did reach the model, and the next question is whether an over-long input ever raises. If it raises, the system fails loudly and you are safe. If it never raises and the vector still moved, you are in one box holding two possibilities: a limit above your input, or a split-and-average wrapper. Those two look identical from outside, which is why they share a box rather than being separated by a branch the test cannot make.

Follow the discarded branch and the binary search gives a cut point, which the last decision compares against the documented limit. A cut point equal to the documented limit means the clipping is happening where you expect and the fix is in your chunker. A cut point below it — 256 against a documented 512, say — means something in the path has its own opinion, and the candidates are an inference server started with auto-truncation at a shorter length, a framework wrapper passing its own max_length, or a provider quietly enforcing less than it publishes. That distinction changes who you talk to: the first two are yours to fix in a config file, and the third is a support ticket and a capacity plan. A cut point above the documented limit points the other way: the deployment is encoding more than its documentation admits, which means the documentation is describing something other than the thing you are running.

The sweep is cheap enough to keep, and the cost is derivable from the code rather than taken on trust. The search starts with a low of 8 and a high of 8192, and each iteration replaces the surviving range with half of it. Because 8192 is a power of two, the count of halvings that collapses that range to a single value is the number of doublings it takes to climb from one to 8192 — which is the middle of the twelve-to-fourteen band in the paragraph below. It is a band rather than a point because the range starts at 8 instead of at zero and the loop stops the moment the two ends meet. Each iteration performs exactly one model.encode, on a pair of strings, so the number of encode calls equals the number of halvings.

Twelve to fourteen encode calls at a few hundred milliseconds each is under ten seconds of wall clock, which fits comfortably in a nightly job or in the integration suite that already talks to a real endpoint. The loop bound in the code and that wall-clock figure are the same fact stated twice, and their agreement is what licenses keeping the sweep in continuous integration rather than running it by hand once a quarter. Written as a test that asserts the measured cut point equals the value in your configuration, it converts a class of silent regression — someone edits a deployment manifest, someone bumps a framework minor version, a provider changes enforcement — into a red build with an obvious diff. The assertion is worth writing against the number rather than a range, because a cut point that moves by even a few tokens means something in the path changed, and you would rather find out from continuous integration than from a recall metric — the share of queries whose supporting text actually comes back from the search — six weeks later.

The scope of a measured cut point#

The test does not measure damage. Proving that 3,400 tokens never reached the encoder — which is what a roughly 3,900-token body reduced to a 510-token cut amounts to — says nothing about whether those tokens mattered for retrieval, and the answer depends on what was in them, which is chapter 3.

It also covers exactly one call path. Most systems embed documents through an indexing job and queries through a serving path, and those are frequently different code — a batch worker using the local model directly and an API handler going through a framework's retriever abstraction, or the reverse. The tail test on the indexing path tells you nothing about the query path, and query-side truncation is easy to miss precisely because queries are usually short. Run it on both, and run it against whatever fixture the CI environment uses if that differs from production.

The sweep gives you a number that is correct for the deployment you ran it against. A hosted endpoint can change its enforcement without changing its documentation, and a self-hosted inference server picks up new flags whenever someone edits the deployment manifest, so the cut point is a measurement with a shelf life rather than a constant. Storing it as a value your ingestion code reads, and re-measuring it on a schedule, is what turns it into something you can rely on — and it is also the number chapter 6's instrumentation has to be calibrated against, because per-chunk token counts checked against a limit taken from the docs will report everything as healthy while a lower provider cap eats the tails.

Two more limits are worth naming before moving on. The test detects truncation, not reordering or normalisation, so a system that strips markdown, collapses whitespace or lowercases the input before encoding will show a stable vector for reasons unrelated to length. And it says nothing about batch-level caps: an endpoint that accepts 512 tokens per item may still silently drop items past the two-hundredth in an array, which is a different failure with the same shape of silence and is worth a separate probe if you batch aggressively.

An over-long input that comes back with an ordinary vector and no error is evidence about the client's error-handling policy, and evidence about nothing else. The capacity question it appears to answer needs two measurements instead. The tail test asks whether the last tokens of an input reach the model, and a pair of identical vectors answers no. The boundary sweep asks where the cut falls, and answers with a number in your own tokens, which the documented limit can then be held against: equal points at your chunker, lower points at a second limit in one of the four places a limit can be set, and higher says the documentation is not describing the thing you are running.

The reason to spend the ten seconds on that rather than on another token count is that a token count cannot move when the deployment moves, and everything expensive here happens on the deployment side. You now have a way to prove that text is disappearing and a number for where the cut falls. The next question is why losing the last third of a chunk costs far more than a third of its usefulness.