A document with zero chunks is missing, not merely unretrieved

An assistant answering from a corpus of a few hundred internal documents will answer a question about a document that is not in its index, confidently, from whichever neighbouring document is nearest. A chunk is the unit that gets stored and retrieved — a passage of a few hundred tokens cut out of a document — and a top-k query is a request for the k chunks whose stored vectors are closest to the question's vector. Nothing in either of those definitions mentions documents that ought to exist, which is the whole of the difficulty: a retrieval system can report which chunks it found and cannot report which ones it never had.

The distinction that costs teams the most time is between a document that is missing and a document that is merely unretrieved. Unretrieved means the chunks are in the index and something ranked above them, which is a relevance problem, and relevance problems respond to a bigger k, a better query, or a reranker. Missing means the chunks were never written, which is an ingest problem, and no retrieval setting reaches a row that does not exist. Telling the two apart takes seconds if a pipeline has written down what each stage produced per document, and takes an afternoon of raising k if it has not.

Absence has no rank#

Retrieval cannot express absence for a structural reason rather than because somebody forgot to add it. A top-k query is a sort followed by a truncation. The sort needs a total order over the stored rows and it has one, since every stored vector has a cosine similarity against the query vector and that number is defined for every row without exception. The truncation takes the first k, and as long as the index holds at least k rows it yields exactly k. The response therefore has the same shape, length and well-formedness whether or not any particular document is represented.

What the index contains is identifiers, vectors and payloads, and no statement anywhere about which documents ought to be present, because nothing ever wrote one. A query is a function of what is stored, so a fact that was never stored cannot appear in an answer. The sentence the 2024 policy is not here is not one the index has the vocabulary to say.

Those two steps together give the result. The system's behaviour when a document is missing is identical to its behaviour when the document is present and merely outranked — a confident answer from the nearest available neighbour, which for a question about a 2024 refund policy is a 2021 refund policy, a document written by the same team in the same house style about the same subject. Raising top-k explores the wrong axis entirely, and it explores it slowly, because each attempt looks almost like it might have worked.

The belief being punished there is worth naming, because it is a reasonable belief and not carelessness. Raising k feels like a defence against having missed something: search harder, look further down the list. What makes it useless here is precisely what makes it feel safe — it behaves identically in the two cases it would have to separate. Ask for five and five come back. Ask for twenty and twenty come back. The response is well-formed, the scores are plausible, and nothing in the shape of the result differs between the build where the document indexed and the build where it produced nothing at all. A check whose output is constant across the distinction it is meant to draw carries no information about that distinction, however much work it takes to run.

The same conclusion arrives from the other direction if you write correctness as a product. An answer that cites the right document requires all of these at once: the document was enumerated by the reader, it extracted to non-empty text, that text produced at least one chunk, the chunk was embedded, the vector was written into the index, the index returned it for this question, and the renderer put it into the prompt. Those factors multiply rather than add, so a zero anywhere makes the product zero. A document that extracts to almost nothing zeroes the third factor, and after that a larger k, a reranker and a better-worded template are all improvements to factors that were already non-zero, multiplying against a zero that none of them can touch. This is why the order of investigation dominates the quality of any individual fix.

Absence does become expressible one stage earlier, and the reason is worth stating plainly: absence is a relation between two artifacts rather than a property of one. A document id that appears in a parse output and produces no rows in a chunk output is absent, and that statement requires both files to be in hand. A single artifact cannot express it — which is exactly why the fix belongs at ingest, where two files can be compared, and not at query time, where only one exists.

Three counters that settle it in seconds#

The fix is a manifest, meaning a file with one row per document, joined across the ingest stages, holding the three quantities that each stage is responsible for producing.

# stage7_manifest.py — reconciles parse, chunk and embed per document
import json
from collections import Counter
from pathlib import Path
import numpy as np

def build_manifest(build: Path) -> list[dict]:
    parsed = {json.loads(l)["doc_id"]: json.loads(l)
              for l in (build / "parse.jsonl").open()}
    chunk_counts: Counter[str] = Counter()
    chunk_tokens: Counter[str] = Counter()
    for line in (build / "chunks.jsonl").open():
        row = json.loads(line)
        chunk_counts[row["doc_id"]] += 1
        chunk_tokens[row["doc_id"]] += row["n_tokens"]

    ids = json.loads((build / "vector_ids.json").read_text())
    vectors = np.load(build / "vectors.npy")
    vector_counts: Counter[str] = Counter(cid.rsplit("#", 1)[0] for cid in ids)

    rows = []
    for doc_id, doc in sorted(parsed.items()):
        rows.append({
            "doc_id": doc_id,
            "bytes_on_disk": Path(doc["path"]).stat().st_size,
            "chars_extracted": doc["chars"],
            "chunks": chunk_counts[doc_id],
            "tokens_chunked": chunk_tokens[doc_id],
            "vectors": vector_counts[doc_id],
            "embedding_dim": int(vectors.shape[1]),
        })
    return rows

Two details in that listing carry the whole argument, and both are easy to read past. The loop iterates parsed, so it emits a row for every document the parse stage saw, and it looks the chunk count up in a Counter, which returns zero for a key it has never seen rather than raising. Those two choices together mean the manifest reconciles without a residual: documents parsed equals documents with chunks plus documents with zero chunks, and there is no third category, because a document cannot fail to get a row. That closing identity is what licenses the whole method. The set of missing documents is not something to go hunting for — it is exhaustively enumerable by a sort, and a set you can enumerate is a set you can fix.

The other detail is cid.rsplit("#", 1)[0], which recovers a document id from a chunk id by cutting at the separator. That works only because the chunker built chunk ids as a document id and an offset joined by a #. Had the ids been opaque UUIDs, the vector count could not be attributed to a document at all without a second lookup table, and the third counter — the one that catches a half-written embedding batch — would not exist. A design decision two stages upstream is what makes this reconciliation possible.

Run over the 412-document corpus and print the rows sorted by chunk count ascending, which puts every problem at the top of the listing.

That table is worth reading as a set of gaps rather than as a set of levels, because none of the individual numbers means anything on its own. The first gap is between the bytes column and the chars column. For the two markdown rows, dividing characters by kilobytes lands almost exactly on a thousand characters per kilobyte, and it lands there twice, on documents of very different sizes — two independent measurements agreeing on one figure, which is what turns a rule of thumb into a property of this corpus. The 2021 policy, a PDF carrying page images alongside its text layer, lands in the middle of a much lower band. The top two rows land three orders of magnitude below even that.

The second gap is between the chars column and the chunks column, and it closes exactly. A shade under four characters to a token fits all three non-zero rows at once: it turns the 2021 policy's characters into a token count that divides into sixteen four-hundred-token windows, turns the billing handbook's into forty-one, and leaves the rota's inside the two it reports. One ratio, three rows, no residual. The chunks column is therefore not independent evidence — it is a prediction from the chars column, so a row whose chunk count departs from what its character count predicts is a filter or a parser doing something rather than a document being unusual.

The two zero rows break that pattern, and the break is the finding. Forty-three characters is nine tokens, and the chunker drops windows under 32 tokens — a deliberate guard added to stop trailing scraps of a document becoming their own chunks. Nine is under thirty-two, so the single window that text produces is discarded, so the document contributes zero chunks. That guard turned a document with almost no extractable text into a document with no presence in the index whatsoever, and did it without a warning, a non-zero exit code, or a single line of output. The arithmetic fully explains the zero, so nothing exotic happened downstream and the entire fault is upstream of the chunker.

The third gap is the one that is not there. Every row in the table has the same number in the chunks column and the vectors column, so one column is a copy of the other, and the copy holds even on the two rows where both are zero. That equality is the invariant the embed stage owes the rest of the pipeline, and the table is the evidence it held on this build. A row where those two columns differ localises the fault to exactly one stage.

The 2024 policy is 491 KB on disk and 43 characters after parsing, which is a ratio no legitimate document produces. That single row replaces the entire top-k investigation, and it makes the answer to what do you look at first unambiguous: the per-document chunk count for the document you expected to be retrieved, because a zero there means the document is not in the index at all and no retrieval setting can reach it. A second document, the forms appendix, has the same signature and nobody had noticed, which is the usual outcome once you look — silent extraction failures cluster, because they come from a shared origin such as a scanner, an export tool or a locked template.

The mechanism behind a near-zero character count is worth one paragraph, because a reader who does not have it will assume the parser broke. A PDF scanned from a printout carries page images and no text layer at all, meaning the characters were never stored in the file alongside the picture of them. There is nothing for an extractor to read, and PdfReader.extract_text() on an image-only page returns an empty string without raising anything. The parse stage therefore succeeds, writes a file, records a row, and exits zero, on a document from which it recovered nothing. Every signal that stage emits is identical to the signal it emits on a document that extracted perfectly.

Three invariants over the manifest catch this class before anyone asks a question, an invariant here being a relation between the columns that has to hold on every build and is asserted rather than assumed. Characters extracted per kilobyte on disk should sit in a band — plain markdown lands near 1,000 chars/KB, a text-layer PDF between 40 and 400 depending on images, and anything under 5 is a document that did not extract. Chunk count should be zero only when the character count is genuinely zero, so a document with more than 200 characters and zero chunks is a filter eating content. And vectors must equal chunks for every document, an equality that holds trivially when everything succeeds and breaks the moment an embedding batch fails halfway and the retry logic writes a shorter array than the id list.

def check(rows: list[dict]) -> list[str]:
    problems = []
    for r in rows:
        density = r["chars_extracted"] / max(r["bytes_on_disk"] / 1024, 1e-9)
        if density < 5:
            problems.append(f'{r["doc_id"]}: {density:.1f} chars/KB — no text layer?')
        if r["chars_extracted"] > 200 and r["chunks"] == 0:
            problems.append(f'{r["doc_id"]}: {r["chars_extracted"]} chars, 0 chunks')
        if r["chunks"] != r["vectors"]:
            problems.append(f'{r["doc_id"]}: {r["chunks"]} chunks, {r["vectors"]} vectors')
    return problems

Two different obligations are bundled in wiring this into a build. The requirement is that every document somebody put in a corpus be answerable from that corpus, which is what the whole system was built to do. Making check() a non-zero exit is a design decision that serves it, and others would serve it too — a report nobody has to act on, a dashboard, an alert. Failing the build turns a class of failure which used to surface as a wrong answer weeks later into a failed ingest that somebody fixes the same morning, and it works by removing the option of not noticing.

For the 2024 policy the fix is an OCR pass, optical character recognition being the step that reads the shapes of characters off a page image and writes them back out as text: running the six pages through Tesseract, a long-established open-source OCR engine, at 300 DPI takes about 40 seconds and yields 21,400 characters, 14 chunks and 14 vectors, and the assistant answers fourteen days from the correct document on the next query. Making that fallback conditional on the density check rather than unconditional matters at corpus scale, since OCR at 40 seconds a document over 412 documents would add four and a half hours to a rebuild that currently takes five minutes, while running it on the two documents that need it adds eighty seconds.

The gap between those two figures is the whole value of the density check, and it is a ratio rather than a saving. Unconditional OCR costs forty seconds times every document; conditional OCR costs forty seconds times the documents that fail the check. The forty seconds cancels exactly, leaving the ratio of the two counts and nothing else, which means the check pays for itself in proportion to how rare the failure is. A corpus where half the documents were scans would get almost nothing from the conditional. This one gets almost everything.

What only the previous build can see#

Absolute thresholds are the weaker half of this. The stronger check compares today's manifest against the last one, because most ingest damage arrives as a change rather than as a state, and a document that used to produce 41 chunks and now produces 3 is a louder signal than any density band.

def regressions(previous: list[dict], current: list[dict]) -> list[str]:
    prev = {r["doc_id"]: r for r in previous}
    out = []
    for r in current:
        old = prev.get(r["doc_id"])
        if old is None:
            continue
        if old["chunks"] and r["chunks"] < old["chunks"] * 0.5:
            out.append(f'{r["doc_id"]}: {old["chunks"]} -> {r["chunks"]} chunks')
    for doc_id in set(prev) - {r["doc_id"] for r in current}:
        out.append(f"{doc_id}: present in previous build, absent now")
    return out

As an analogy, a manifest is a stock count: walk the shelves and write down what is on them. It breaks in the direction that understates the problem. A warehouse compares its count against a list of what it ordered, so it knows what it is missing, whereas an ingest run has no such list — nothing anywhere records how many chunks a document ought to produce. That absent list is why the strongest check here compares one build against another rather than against an expectation.

The second loop catches the failure nobody instruments for, which is a document that disappears from the corpus entirely — deleted in a cleanup, moved into a directory the glob does not match, or renamed with an extension the parser skips. That produces no row anywhere, no error, and a corpus that is quietly smaller than the team believes, which is why the check has to be written against the previous build rather than against the current one. A document that produces no row is invisible to any check that iterates rows, so the only way to see it is to iterate a set that still contains it, and the previous manifest is the only such set available. Storing each build's manifest as build/manifest-<index_version>.jsonl costs a few hundred kilobytes and makes the comparison a two-file diff.

Reconciling from artifacts rather than from reports#

There is a reason the manifest is built by reading the stage outputs back rather than by having each stage report its own numbers as it runs. A stage that reports its own count is reporting what it believes it did, and the two failures that matter most — a write that did not land, a filter that silently discarded rows — are exactly the cases where belief and file diverge. Reconciling from the artifacts means the chunk count comes from counting lines in chunks.jsonl and the vector count comes from the shape of vectors.npy, so the manifest is a statement about what exists rather than about what the code intended. It also means the manifest can be rebuilt for any past index version without rerunning anything, which is what makes the previous build available for comparison.

The failures each counter can and cannot see#

Counters are cheap and each one has a narrow beam. Knowing which failure produces which signature is what makes the manifest a diagnostic rather than a dashboard.

Read the third column on its own and the first two rows say the same thing twice, which is informative rather than redundant: a scanned PDF and an encrypted one are indistinguishable at query time, and the manifest separates them only because one leaves a near-zero character count and the other an exact zero. Read the second column and every row but the last names a comparison between two numbers that ought to agree — and the last row breaks the pattern by naming no comparison at all.

The last row is the honest limit. A scanned page run through a bad OCR pass yields 18,000 characters of ligature soup, twelve chunks and twelve vectors, and every counter in the manifest reports a healthy document. Nothing structural distinguishes it from a good extraction, so the only detection is reading the text — which is why the parse stage writes build/text/*.txt at all, and why a two-minute habit of opening the extracted text for any document you are about to rely on is worth more than another invariant. A cheap approximation is a dictionary-word ratio over the extracted text: real English prose runs above 0.85, and OCR garble on the same corpus came in at 0.31, which catches the worst of it without pretending to be a quality metric.

Timings, and the counters that live at query time#

Per-stage timing belongs in the same manifest because it is the only way to know where a rebuild goes. Over the 412-document corpus:

{"stage": "parse",    "docs": 412,  "seconds": 71.4,  "notes": "58s in pypdf"}
{"stage": "chunk",    "chunks": 9842, "seconds": 6.2}
{"stage": "embed",    "chunks": 9842, "seconds": 214.0, "batches": 77, "retries": 2}
{"stage": "index",    "vectors": 9842, "seconds": 1.1}

Those four lines close against a figure stated elsewhere in this chapter, and the closure is worth checking rather than assuming. Added together the four stage timings come to a little under five minutes, which is the rebuild time quoted in the OCR arithmetic above. Two independent statements landing on the same number licenses something specific: the four stages account for essentially all of the wall clock, so there is no fifth cost hiding between them. Any future rebuild that takes materially longer than the sum of its own stages is spending that time outside the stages, and that is a different investigation.

The chunk count also closes across three of the four lines. Chunk, embed and index all report 9,842, which is the same quantity travelling through three stages without loss. The parse line reports documents instead, because it is the one stage whose input and output units differ. Where those three numbers stop agreeing, the stage where they first diverge is the stage that lost rows.

Embedding dominates at 214 seconds, which sets the cost of any change that invalidates every vector and is a number worth having before you propose one. Parse taking 71 seconds is also informative, since 58 of those are one library on PDFs, and it tells you that adding an OCR fallback for six documents will not meaningfully change the rebuild.

Timings recorded per stage rather than per run also make one specific pathology visible. The embed stage reports 77 batches and 2 retries, and a retry count that climbs across builds means the provider is rate-limiting you, which on most client libraries is handled by backing off and, on some, by returning fewer embeddings than inputs. That is the failure the chunks-equals-vectors invariant catches, and the retry counter is what tells you it is coming before it does. Both are worth recording because they move at different times: the retry count moves before any document is damaged, the count mismatch only after.

Query time has its own counters, and the same principle applies: emit the quantity, not the impression. The three worth having from the first day are the top-1 and top-k similarity scores, the rendered prompt's token count, and the milliseconds spent in each of embed-query, search and generate.

{"q": "what is the refund window under the 2024 policy?",
 "top_scores": [0.612, 0.594, 0.581, 0.577, 0.566],
 "prompt_tokens": 2144, "answer_tokens": 88,
 "ms": {"embed": 96, "search": 4, "generate": 1380}}

That one line already carries two useful facts. Search costs 4 ms against a 9,842-row matrix, so nothing about latency will ever justify weakening retrieval quality at this scale, and the 96 ms spent embedding the query is twenty-four times the search itself. Prompt size at 2,144 tokens for top-5 sets the budget arithmetic for later: five 400-token chunks plus instructions is roughly what you would predict, and a number that comes in far under prediction means a chunk was dropped somewhere between retrieval and rendering.

That prediction has a residual worth naming rather than rounding past. Five chunks at four hundred tokens each is two thousand tokens, and the recorded figure sits somewhat above it; the difference is the template's instruction line, the question, and the bracketed chunk-id header written above each block. The score list closes more simply still: a top-5 query logged exactly five scores, so the list is complete, and one that logged four would mean a chunk went missing between the retriever and the log.

Where a counter is silent by construction#

embedding_dim earns its place in the row for a reason that has nothing to do with debugging a single document. It is the field that catches an index built from two different models, which happens whenever someone changes the model constant and re-runs the embed stage over only the new documents. Cosine similarity between a text-embedding-3-small vector and a text-embedding-3-large vector is not merely wrong, it is undefined at the shape level and will raise, which is the good case; the bad case is two models of the same dimension, where the arithmetic succeeds and the scores are meaningless. Recording the model name alongside the dimension, per batch, is one more string and closes that hole.

That bad case has the same shape as the top-k defence this chapter opened on, one layer down. A dimension check varies with the thing it must discriminate only when the two models happen to differ in width, and two models of the same width leave it constant — a healthy number on the build where the vectors agree and on the build where half of them mean nothing. The model name closes the hole because it varies whenever the thing being discriminated varies, which is the property to demand of any check before trusting it.

The comparison against the previous build has a blind spot of its own. It can only report change, so a corpus that was already broken on the first build it ever ran looks perfectly stable forever, and nothing in regressions() fires on a document that has produced zero chunks since the beginning. That is why the absolute thresholds stay in place alongside it rather than being replaced by it: the two checks cover each other's gaps and neither covers its own.

The density bands deserve the same scoping. They are the observed ranges for one corpus of markdown files and text-layer PDFs, and a corpus of image-heavy scanned material would sit legitimately low against them, so the numbers are not portable as written. The method is: measure the bands on your own corpus first, put each threshold in the empty space between two observed ranges rather than inside one, and treat a document that crosses one as a different kind of thing rather than an unusual example of the same thing. A reviewer should ask to see the measurement behind a threshold before believing any alert built on it.

Counters make absence visible at ingest, and the query-time line above is where that stops being enough. It is a log statement, which means it is a string in a stream mixed with every other request, tied to nothing that identifies the query it came from, and reconstructing which chunks produced which answer means correlating by timestamp and hoping the two lines were not interleaved with another user's. A counter that cannot be joined back to the question that produced it answers a question about the corpus and none about the answer.

The distinction this chapter turns on holds in both directions. A document with zero chunks is missing, and every retrieval-side remedy is multiplying against a zero; a document with chunks that never surfaces is unretrieved, and retrieval-side remedies are exactly the right tools. Three numbers per document — characters, chunks, vectors — decide which of those two you are in, and they decide it before anybody asks a question rather than weeks afterwards. The next chapter is about the artifact that has to exist per query, because for a wrong answer the question is not how many chunks the document produced but which exact string the model was handed, and that string cannot be recovered from any counter.