Embedding input limits and silent truncation

Read these first

• Embedding models

What you will know

• Measure the model's maximum sequence length and record whether the API counts tokens, bytes, or characters

  The limit is tokenizer-specific, so a character-based chunking rule can exceed it unpredictably across languages and versions.

• Prove whether your library truncates silently

  A passing boundary test should distinguish an exception, clipping, and a provider-side cap; wrappers often hide the last two.

• Show a chunk longer than the limit losing its tail

  Tail loss is particularly damaging when headings or conclusions carry the distinctive terms used by retrieval evaluation.

• Run the same retrieval evaluation with chunks below and above the embedding limit, and compare recall

  A controlled size sweep separates truncation damage from changes in overlap, corpus coverage, or query quality.

• Instrument indexing to report token counts, truncation events, and retained-token ratios for every chunk

  These fields turn an invisible preprocessing defect into an alertable data-quality signal before the index reaches production.

Chapters

1. 1

   The limit is counted in the model's tokens, not your characters

   Maximum sequence length is expressed in the model's own tokens, so a 2000-character rule sold as "roughly 500 tokens, safely under 512" is a guess that fails hardest on source code and configuration, where punctuation and identifiers explode into tokens. Establish the real limit and the unit the API counts in, then account for what quietly eats the budget: special tokens, and any prefix or metadata you prepend.

   Read chapter
2. 2

   A client that accepts an over-long chunk has already discarded text

   Handing a 900-token chunk to a 512-token model normally produces a perfectly ordinary vector and no warning whatsoever. Three behaviours hide behind that silence — a raised exception, client-side clipping, or a provider cap below the documented one — and reading the documentation separates none of them. Embed a long text, embed it again with a distinctive tail appended, and compare: identical vectors mean the tail never reached the model. That proves loss but not its source, so finish with a boundary sweep — grow the input a little at a time and find the length at which the vector stops changing. That cut point is the effective limit, and holding it against the documented one is what tells clipping apart from a provider cap set lower than advertised.

   Read chapter
3. 3

   Truncation removes the tail, where the conclusion lives

   Clipping always takes the end, and the end is where resolutions, closing summaries and the distinctive terms an evaluation query uses tend to sit, so losing thirty percent of a chunk costs far more than thirty percent of its retrievability. Build a document whose only distinguishing vocabulary appears in its final paragraph, index it over the limit, and watch the query that should find it return nothing.

   Read chapter
4. 4

   One pooled vector gets blurrier as the passage it summarises gets longer

   Truncation is not the only reason long documents retrieve badly, which is why fixing it does not close the gap. A single fixed-length vector has to stand for everything in its input, so a long passage spanning several subjects averages into a point that matches each of them mediocrely, while a short focused passage sits directly on top of its own topic. Split the long document instead and its evidence spreads across many chunks, so the one sentence that answers the query is pooled with a page of surrounding material and no single chunk scores strongly. This is dilution, and it is a different defect from truncation: every token reached the model and the match is still weak.

   Read chapter
5. 5

   A chunk-size sweep separates truncation damage from chunking effects

   Re-run one retrieval evaluation with chunks sized just under the limit and again comfortably over it, holding overlap, corpus coverage and query set fixed so the delta is attributable to truncation alone. Predict the direction and rough size before running it. Then face the residue with a name for it: long-document recall improves without reaching parity with short documents, and dilution is the part truncation never explained.

   Read chapter
6. 6

   Token counts and retained ratios make an invisible defect alertable

   Emit four fields for every chunk at index time: its token count, the model's effective limit, whether truncation occurred, and the retained-token ratio. Together they convert a preprocessing defect that no exception ever reports into something a dashboard shows and an alert fires on, well before the index reaches production. You will also decide where in the pipeline to measure and which threshold deserves a page.

   Read chapter