Six inspectable boundaries, not one convenience call

A language model's weights were fixed before your company edited its escalation procedure last March, so the weights contain nothing about it, and asked who to page the model will still produce fluent sentences with nothing behind them. Retrieval-augmented generation closes that gap by searching a prepared copy of an organisation's documents at the moment somebody asks, pasting the passages it finds into the prompt, and telling the model to answer from them. Putting the relevant paragraphs into the prompt is the cheapest way to make that text available at the moment the question arrives. The plainest form of the arrangement is what this topic means by naive RAG: the question is turned into a vector, the handful of passages nearest it come back from an index, those passages are pasted into a prompt template, and a single model call answers from them. No step rewrites the question, reranks the results, or goes back for a second round of searching. Most engineers meet it in something close to that form — an internal assistant over runbooks and policies, a support bot over a help centre, the "chat with your documents" quickstart in a vendor tutorial — because that is what a first version looks like when a team has an afternoon and a directory of markdown files.

The usual way to get there in an afternoon is a framework: a library such as LangChain or LlamaIndex that packages reading, splitting, embedding, storing, searching and prompting behind a few calls, with a default chosen for every decision it makes on your behalf. The rest of this topic builds the same pipeline without one. Hand-written stages do not retrieve better than a maintained library's, and the thing they change is the cost of an explanation: when a stage's choices and its outputs both live inside somebody else's call, explaining a wrong answer takes days rather than minutes, as the incident worked at the end of this chapter illustrates. A wrong answer is a claim about a chain, and a chain can only be searched at the points where its intermediate values can be read.

Why a boundary is worth a file#

Take the wrong answer apart into the chain that produced it. The answer is a function of the prompt string and nothing else. The prompt string is a function of the retrieved subset and the template. The retrieved subset is a function of the index and the query vector. The index is a function of the vectors. The vectors are a function of the chunk texts. The chunk texts are a function of the extracted text. The extracted text is a function of the bytes on disk. Each arrow there is a function of its input alone, which is the property everything below rests on: if the value going into a stage is right and the value coming out is wrong, the fault is inside that stage and cannot be anywhere else.

That property is only usable if the values can be read, so the ability to localise a fault is not a property of the pipeline's design. It is a property of which intermediate values happen to have been written down. A stage whose output exists as a Python object inside a call that has already returned offers no comparison point at all.

Given an ordered chain and a readable value between every pair of stages, finding the fault is a bisection. Read the value in the middle and decide whether it is what you expected. If it is, everything upstream is exonerated and the fault lies in the second half; if it is not, the first half holds it. Halving six candidates three times leaves one. A call that spans four stages exposes what went into it and what came out and nothing between, so the number of fault locations you can distinguish behind it is one. There is nothing to halve, and the only method left is to read the implementation and reconstruct by hand the values it did not keep, which is what turns minutes of work into days.

The same chain read forwards rather than backwards gives the other half of the argument. A correct answer requires every one of these to hold at once: the document was found by the reader, it extracted to real text, that text produced at least one chunk, the chunk was embedded, the vector was written into the index, the index returned it for this question, and the renderer put it into the prompt string. Correctness is a product of those factors rather than a sum of them, and a product with a zero anywhere in it is zero. A document that extracts to nothing contributes zero at the second factor, and after that a better embedding model, a larger top-k and a more carefully worded template are all multiplying against zero. This is why the order of investigation matters so much more than the quality of any single fix: improving a factor that is already non-zero cannot rescue an answer that is being killed by a factor that is zero.

By way of analogy, a convenience call is a sealed relay: a parcel goes in, something comes out, and if it arrives at the wrong address no postmark says which leg carried it astray. The analogy breaks in the direction that understates the problem, because a postal system at least still has the parcel. A chain assembled inside one process destroys each intermediate the moment the next stage consumes it, so the evidence has not been mislaid — it has been freed.

The eleven lines below are a complete naive RAG pipeline, and the shape in which most teams first meet all six changes of representation at once.

from langchain.indexes import VectorstoreIndexCreator
from langchain_community.document_loaders import DirectoryLoader

loader = DirectoryLoader("corpus/", glob="**/*.md")
index = VectorstoreIndexCreator().from_loaders([loader])
print(index.query("who do I page for a database failover?"))

Every intermediate in that listing is discarded by construction rather than by accident. The loader's extracted text is returned as objects and never written to disk. The chunks live inside an in-memory vector store that is rebuilt from scratch on process start, so a restart erases the index that produced yesterday's answer. The prompt string is assembled inside the chain, sent to the model, and dropped. None of that is a defect in the library — an object that nobody asked to be persisted is correctly not persisted — and all of it means the chain has exactly one readable boundary, which is the question going in and the string coming out.

Six stages, six files#

Work over a small corpus first, small enough that you can read the intermediates end to end. Twelve documents is a good size: a handful of policy markdown files, two runbooks, a product handbook exported as PDF, and a support FAQ. The pipeline reads corpus/ and writes everything else under build/, and each stage is a separate script that reads the previous stage's file and writes its own.

The generate step is on the diagram for completeness and is barely code — one API call taking the contents of the prompt file. The six stages before it are where every decision that determines the answer gets made, and each one leaves behind an artifact whose absence or contents can be checked without rerunning anything upstream.

Topic 9.1 described this pipeline as a block diagram, with arrows from documents to chunks to vectors to an answer, and stopped there. The exercise here is to make each of those arrows a file on disk. None of the code that follows is better than what a library would do — each stage is twenty to forty lines that a maintained package does more carefully, with more file formats, more retries and better batching. The difference is not quality of implementation. It is that when a stage writes its output to a path you chose, the output is still there tomorrow, and the question of what the model actually saw is answered by opening a file rather than by attaching a debugger to somebody else's abstraction.

Parsing is the stage most people skip past, and it is the one that fails most silently. Extraction from PDF is lossy in ways that depend on how the PDF was produced, so the honest version of this stage writes the extracted text out where you can read it.

# stage1_parse.py
import json
from pathlib import Path
from pypdf import PdfReader

SUFFIXES = {".pdf", ".md", ".txt"}

def extract(path: Path) -> str:
    if path.suffix == ".pdf":
        return "\n".join(page.extract_text() or "" for page in PdfReader(path).pages)
    return path.read_text(encoding="utf-8")

def main(corpus: Path, build: Path) -> None:
    (build / "text").mkdir(parents=True, exist_ok=True)
    with (build / "parse.jsonl").open("w") as manifest:
        for path in sorted(p for p in corpus.rglob("*") if p.suffix in SUFFIXES):
            doc_id = path.relative_to(corpus).as_posix()
            text = extract(path)
            flat = doc_id.replace("/", "__") + ".txt"
            (build / "text" / flat).write_text(text, encoding="utf-8")
            manifest.write(json.dumps({"doc_id": doc_id, "path": str(path),
                                       "chars": len(text)}) + "\n")

Two separate things are being decided in those twenty lines. The requirement is that a wrong answer be explainable, which follows from the chain argument above and is not negotiable. Writing the extracted text to build/text/*.txt under a flattened document id is a design decision that satisfies it, and others would satisfy it too — a column in a database, a single tarball, a log line per document carrying the whole extraction. A file per document wins here because it is the form a person can grep and a diff can compare.

Chunking is where the retrieval unit is defined, and the choice of unit constrains everything downstream, because the smallest thing the retriever can return is one chunk and the smallest thing it can miss is also one chunk. A fixed token window is the simplest defensible starting point, and writing it yourself takes ten lines.

# stage2_chunk.py
import json, tiktoken
ENC = tiktoken.get_encoding("cl100k_base")

def windows(text: str, size: int = 400, overlap: int = 0):
    ids = ENC.encode(text)
    step = size - overlap
    for start in range(0, len(ids), step):
        window = ids[start:start + size]
        if not window:
            break
        yield start, ENC.decode(window), len(window)

def main(build):
    with (build / "chunks.jsonl").open("w") as out:
        for line in (build / "parse.jsonl").open():
            doc = json.loads(line)
            flat = doc["doc_id"].replace("/", "__") + ".txt"
            text = (build / "text" / flat).read_text(encoding="utf-8")
            for start, chunk, n_tokens in windows(text):
                out.write(json.dumps({
                    "chunk_id": f'{doc["doc_id"]}#{start}',
                    "doc_id": doc["doc_id"],
                    "token_start": start,
                    "n_tokens": n_tokens,
                    "text": chunk,
                }) + "\n")

A chunk id that encodes its document and its offset — runbooks/db-failover.md#1200 — costs nothing to construct and pays for itself the first time a retrieval log tells you that rank 1 came from token 1200 of a document you can then open at that offset. Opaque UUIDs for chunk ids are the default in most vector stores, and they turn every retrieval log into a lookup against a second system. The skipped step is short: an identifier derived from a position can be inverted back to that position, and an identifier drawn from a random source cannot be inverted at all, so it has to be looked up wherever the mapping is stored. One design keeps the debugging loop inside a text editor; the other requires the store to be up.

The output of that stage is a file you can read with head, which is worth doing once before trusting anything downstream. Its format is line-delimited JSON, one complete JSON object per line and conventionally given a .jsonl extension, which is what makes every intermediate in this pipeline appendable without rewriting the file and readable a record at a time without a parser:

{"chunk_id": "runbooks/db-failover.md#0", "doc_id": "runbooks/db-failover.md", "token_start": 0, "n_tokens": 400, "text": "# Database failover\n\n## Change log\n\n2024-03-11 — updated primary contact after..."}
{"chunk_id": "runbooks/db-failover.md#400", "doc_id": "runbooks/db-failover.md", "token_start": 400, "n_tokens": 400, "text": "...| Region | Primary | Secondary |\n| eu-west-1 | A. Haddad | ..."}

Those two lines are worth reading as a pair of gaps rather than as two records, and the gap that matters is between what a heading says a chunk is about and what the chunk contains. The first record starts at token 0 and opens on a document title and a change log, so a model asked to embed it is embedding a passage about revision history. The second starts at token 400 and opens on a table row, with no heading anywhere above it inside the chunk, so the passage that actually holds the current on-call names arrives at the embedding model carrying no statement of what it is about. Neither record is malformed. The document was split at a fixed offset, the offset happened to fall after a change log, and the names a question would ask for now sit in the one chunk whose text gives an embedding model the least help in finding them.

The change log occupying the first four hundred tokens, and the on-call table starting at token 400 with no heading above it, are both facts you can see without running a query, generating an answer or calling a model. They are visible here because the chunker wrote to a path rather than into an object.

Embedding and indexing are two stages rather than one because they fail differently. Embedding is a network operation with rate limits, partial failures and a cost per token; indexing is arithmetic on an array that either has the right shape or does not.

# stage3_embed.py + stage4_index.py, condensed
import json, numpy as np
from openai import OpenAI

client = OpenAI()
MODEL, DIM, BATCH = "text-embedding-3-small", 1536, 128

def embed_all(build):
    rows = [json.loads(l) for l in (build / "chunks.jsonl").open()]
    vectors = np.zeros((len(rows), DIM), dtype=np.float32)
    for i in range(0, len(rows), BATCH):
        batch = rows[i:i + BATCH]
        resp = client.embeddings.create(model=MODEL,
                                        input=[r["text"] for r in batch])
        for j, item in enumerate(resp.data):
            vectors[i + j] = item.embedding
    np.save(build / "vectors.npy", vectors)
    (build / "vector_ids.json").write_text(
        json.dumps([r["chunk_id"] for r in rows]))

def build_index(build):
    vectors = np.load(build / "vectors.npy")
    norms = np.linalg.norm(vectors, axis=1, keepdims=True)
    unit = vectors / np.clip(norms, 1e-9, None)
    np.savez(build / "index.npz", unit=unit,
             ids=np.array(json.loads((build / "vector_ids.json").read_text())))

Normalising the rows once at index time means retrieval is a single matrix multiply and the resulting scores are cosine similarities, with no ambiguity about whether a distance is being reported instead. That ambiguity is worth eliminating deliberately, because a store that returns L2 distance where you expected cosine similarity produces rankings that look plausible and are ordered the wrong way round on some queries. The step under that claim is a sort direction. A similarity is sorted descending and a distance ascending, so a sort written for one and fed the other returns the least relevant results in a shape indistinguishable from the most relevant ones — nothing raises, nothing logs, and the top of the list is full of plausible passages.

Retrieval and rendering close the loop. Retrieval writes what it found and the scores it found it at; rendering writes the exact string it will send.

# stage5_retrieve.py + stage6_render.py, condensed
TEMPLATE = """Answer the question using only the context blocks below.

{context}

Question: {question}
Answer:"""

def retrieve(build, question, k=5):
    idx = np.load(build / "index.npz", allow_pickle=True)
    q = np.array(client.embeddings.create(model=MODEL,
                                          input=[question]).data[0].embedding,
                 dtype=np.float32)
    q /= np.linalg.norm(q)
    scores = idx["unit"] @ q
    top = np.argsort(-scores)[:k]
    return [{"chunk_id": str(idx["ids"][i]), "score": float(scores[i])}
            for i in top]

def render(hits, chunks_by_id, question):
    context = "\n\n".join(
        f'[{h["chunk_id"]}]\n{chunks_by_id[h["chunk_id"]]}' for h in hits)
    return TEMPLATE.format(context=context, question=question)

Running this over the twelve-document corpus produces build/chunks.jsonl with 341 lines, a vectors.npy of shape (341, 1536) weighing 2.0 MB, and a prompt file of about 2,100 tokens for a top-5 query. Every one of those numbers is checkable with a command, and each of them is a number you will want later.

Two of those three close exactly against each other. The array holds one row per chunk and one component per dimension, every component is a float32, and a float32 occupies four bytes; multiply row count by dimension by four bytes and the answer is the two megabytes the file weighs. Given any two of the three quantities the third is determined, so a vectors.npy whose size does not match its own shape is not the array the code believes it wrote. The identity also converts a shape mismatch into a size mismatch, visible from a directory listing without loading anything.

The prompt size closes the same way, with a residual worth naming rather than rounding away. Five chunks at four hundred tokens each is two thousand tokens of context, and the prompt file comes in a little above that. The difference is the template's instruction line, the blank lines between blocks, the question itself, and the five bracketed chunk-id headers the renderer writes above each block. Naming the residual is what makes the number usable: a prompt far under the prediction has lost a context block between retrieval and rendering, and one far over it has picked up something the template did not intend.

What makes a stage boundary worth having is that the representation changes across it. Parsing turns bytes into text, chunking turns one text into many, embedding turns text into numbers, indexing turns a list of vectors into a searchable structure, retrieval turns a question into a ranked subset, and rendering turns a subset back into text. Each of those transitions can fail in a way the others cannot, and each produces a value in a different form, which is why a file per boundary is not arbitrary bookkeeping — it is one artifact per representation. A helper that combines two adjacent stages is combining two representations, and the intermediate one simply never exists.

Wiring the stages together is best done by something that understands file dependencies, because then a change to one stage rebuilds only what depends on it. A Makefile with six targets does this in twenty lines, and it enforces the boundaries socially as well as mechanically: editing the chunker and running make re-embeds, editing the template and running make does not, so the difference in cost between the two kinds of change is visible in how long you wait. That difference turns out to matter a great deal later, when a prompt change can be tried six times in an afternoon and a chunking change cannot.

The generate stage is deliberately trivial — read the prompt file, call the model at temperature 0, write the response — and temperature 0 is not an aesthetic choice. Temperature is the decoding setting that controls how much randomness goes into choosing each token, and at 0 the model takes its highest-probability continuation every time instead of sampling among the plausible ones, which is what makes every experiment in the following chapters a comparison rather than a sampling exercise. With a fixed index, a fixed prompt and that greedy decoding, running the same query twice is expected to produce the same string. Providers do not guarantee bitwise determinism even at temperature 0, but the variation is small enough that a changed answer is evidence something upstream moved.

What a single call swallows#

The value of the split is easiest to see by asking, stage by stage, which question the artifact answers and what happens to that question when the stage has no output of its own.

Read that table down the fourth column rather than across, and the same word keeps appearing — silently, default, dropped, in memory. Every row of that column describes an outcome identical whether the stage worked or not, which is why a missing artifact does not make its question harder to answer. It makes the question unanswerable.

The failure mode this table describes is not that a library is wrong. It is that a call spanning three stages gives you one place to look when something breaks and three places where it could have broken, and no way to tell them apart without instrumenting code you do not own. The exercise is void as soon as one call crosses a boundary — VectorstoreIndexCreator crosses four — because the boundaries are the entire point.

The belief worth naming here is comfortable rather than careless. A framework's stages are individually well tested, so the reasoning goes, and a well-tested stage will not silently produce nonsense. Both halves are true and the conclusion does not follow, because the failures in this chapter are not stage failures. A chunker splitting at a fixed offset has done exactly what it was tested to do when it puts an on-call table in a chunk that reads as a change log, and a loader returning strings has done exactly what it was tested to do when it declines to write them to disk. The fault lives in the composition, and no test of a part examines the composition. The tell is the shape to look for everywhere in this topic: the process exits zero on the night every document indexed cleanly and on the night a third of them extracted to nothing, because on both nights nothing threw. An exit status that is constant across the two outcomes it would have to separate is not a weak signal about ingest health. It is no signal at all.

What a full rebuild costs#

There is a useful lower bound on how much this costs. The whole pipeline above is roughly 140 lines of Python, embedding 341 chunks against text-embedding-3-small at $0.02 per million tokens costs about 0.3 cents, and a full rebuild over twelve documents runs in under twenty seconds, most of it waiting on the embedding API. Scaling to 400 documents brings that to about 9,800 chunks, 3.9 million tokens of embedding at roughly 8 cents, and a rebuild of four to six minutes. Those figures matter because they set how freely you can rerun the whole thing, which turns out to be the thing that makes later chapters possible.

Each of those costs is a product, and knowing the factors is what lets you predict the next case rather than quoting this one. Embedding cost is chunk count times tokens per chunk times price per token. At the small size that is three hundred and forty-one chunks of four hundred tokens, a few hundred thousand tokens, and a few hundred thousand tokens at two cents a million is the fraction of a cent quoted above. At the larger size the chunk count is measured rather than extrapolated from twelve documents, and multiplying it by the same four hundred tokens gives the 3.9 million, which at the same price per million gives the 8 cents. Change the chunk size and every one of those numbers moves in a way you can compute before running anything.

The line count closes against its own claim at the bottom of the band rather than the middle: six stages at twenty to forty lines each predicts a range, and 140 lines sits at the low end. Two pairs of stages are shown condensed into shared files, and a stage sharing a file with its neighbour repeats neither the imports nor the path handling, so the break says the range was mostly measuring per-stage overhead.

Adopting a framework after the stages exist#

None of this is an argument against frameworks, and the last chapter of this topic is about comparing your build against one honestly. A framework that you adopt after having built the stages yourself is a set of defaults you can name; adopted before, it is a set of defaults you inherit. The distinction is about sequence rather than about tooling.

The mechanism behind that distinction is worth stating, because "know your defaults" on its own is advice rather than an explanation. A default stays invisible for as long as you have no expectation to compare it against. Having written a chunker with a four-hundred-token window and no overlap, you hold a specific expectation about how many chunks a document of a given length produces, and a framework whose splitter produces a different number is conspicuous; without that expectation the same number is just a number. The hand-built stage manufactures the expectations that make the framework legible.

Ten thousand chunks, and ten million#

This is also not a production architecture. There is no incremental ingest, so changing one document means re-embedding all of them. There is no persistence beyond files on a laptop, no concurrency control, no access filtering, and no handling of the case where two processes rebuild the index at once. A flat numpy matrix scanned with a matrix multiply is exact and entirely adequate at 10,000 chunks, where a top-5 query takes about 4 ms; at 10 million chunks the same scan takes several seconds and an approximate index becomes mandatory, at which point recall becomes a tunable and the scores stop being exact. Everything in this topic assumes the exact version, because the point is to see the mechanism rather than to serve traffic.

The jump from milliseconds to seconds is worth deriving, because the derivation is what tells you where your own corpus sits. A full scan compares the query against every stored row, so its cost is proportional to the number of rows. Multiplying the row count by a thousand multiplies the time by a thousand, and a few milliseconds multiplied by a thousand is a few seconds. There is no cliff anywhere in that curve. What changes at the larger size is what a user will tolerate rather than the arithmetic, which is why the switch to an approximate index is a product decision wearing an engineering costume.

Building by hand does not make a pipeline correct. Six inspectable boundaries with nothing recorded at any of them is still guesswork, only with more of your own code to blame. Every check described here is one somebody has to remember to run, on a file somebody has to remember to open, about a document somebody has already decided to be suspicious of — and that last condition binds. The method assumes a person who already suspects a specific document, and offers nothing to a person who does not yet know which document to suspect.

The artifacts exist now, but nothing counts them, and a document can pass through all six stages and contribute exactly zero chunks to the index while corpus/ still lists it, parse.jsonl still has a row for it, and every command exits zero. Each of those three signals is constant across the case where the document indexed and the case where it vanished, which by the argument above makes all three worthless for telling the two apart. That gap is what the next chapter closes, by making each stage report what went in and what came out, so that absence becomes a number in a column rather than something a person has to think to look for.