Show the traversal, then run flat retrieval at the same budget

The demo goes well. The graph answers the Shenzhen exposure question, names Aurora and Aurora Lite, and shows the chain. The flat system, asked the same question, returns the risk-management policy and a generic answer. Then the reviewer says: how do you know flat retrieval could not have got there with a better prompt, or a reranker, or a larger k?

That is the correct question and it has an experiment rather than an argument as its answer. The demonstration that settles it for a single question has two halves, and skipping the second half is why most GraphRAG demos prove nothing.

Three terms in that scene carry the argument and are worth pinning down before it starts. A traversal is the walk a graph-based system makes from the entities a question names, out along edges — stated relationships between two things — to the entities those connect to, gathering the documents behind each edge as it goes. Flat retrieval is the alternative it is being compared against: score every chunk in a corpus against the question, take the best k of them, and hand those to the generator. And k is that count, the number of chunks retrieved. The Shenzhen question is the worked example this topic has been building on, where a fire in a Shenzhen industrial park connects to two product lines called Aurora and Aurora Lite through a supplier and a part, across three documents that do not mention one another.

Settling it for one question#

The first half is the traversal, in the form chapter 2 established: intermediate nodes, and for each edge, the document that asserted it and the sentence it was asserted by. What matters for the reviewer is that the chain is inspectable, so that "the graph got it right" becomes a claim they can check in a minute rather than a verdict they have to accept. If a link in the chain has an expired validity window or an OCR-derived provenance record, the reviewer should see that too and discount accordingly.

The second half is running flat retrieval on the same question at the same context budget, and reporting what it retrieved.

graph_ctx = build_graph_context(question)          # traversal + evidence quotes + snippets
budget    = count_tokens(graph_ctx)                # 6,180 tokens on this question

k = max_k_within(budget, chunk_tokens=512)         # 12
flat_hits = reranker.rerank(question, retriever.search(question, top_k=200))[:k]
flat_ctx  = "\n\n".join(h.text for h in flat_hits)

# Same generator, same system prompt, same temperature. Only the context differs.
graph_answer = generate(question, graph_ctx)
flat_answer  = generate(question, flat_ctx)

# The decisive measurement: is the linking fact present at all?
linking_facts = ["Hanyu Precision was a tenant of Building C",
                 "Hanyu Precision is the sole source of CN-4471",
                 "CN-4471 is used in the Aurora mainboard"]
coverage = [any(fact_supported(f, h.text) for h in flat_hits) for f in linking_facts]
# [True, False, False]

The budget line is the part that makes the comparison mean anything, and the arithmetic under it is worth seeing. Isolating retrieval structure as the only difference between the two arms is the requirement; matching on token count is the design decision taken to satisfy it. Once the graph context has been built and measured at 6,180 tokens, the largest k that fits at 512 tokens a chunk is the largest whole number of chunks not exceeding that budget, which is 12 — thirteen chunks would overrun it by several hundred tokens, and the twelve that fit leave a few dozen tokens of slack. That leftover is the residual, and it favours the flat arm by a shade rather than against it, which is the direction a fair comparison wants.

Give the flat arm a smaller budget and the experiment measures token count instead of retrieval structure, and the graph wins for a reason nobody cares about. Give it a larger one and the same thing happens in reverse. Hold the generator, the system prompt and the temperature fixed for the same reason: every one of those is a variable that can move an answer, and an experiment with two variables moved has measured their sum.

That last line of the code is the evidence. It is not that flat retrieval produced a worse-sounding answer, since a fluent wrong answer and a fluent right answer are hard to tell apart at a glance. It is that two of the three facts the answer requires were absent from the flat context, so no prompt over that context could have produced a supported answer, and no combination of the twelve chunks holds the missing links.

The step under that sentence is the one to be sure of, because everything downstream leans on it. A supported answer is one every part of which follows from something in front of the model. The Aurora conclusion needs three facts joined in a chain, and a chain with a link missing does not reach its end — knowing that the fire hit Building C and that the Aurora mainboard uses CN-4471 says nothing about whether Aurora is exposed, unless something also says Hanyu Precision is the part's sole source. So support is a property of the context, not of the generator. A model handed a context with two of the three links missing can still emit the right product names from what it knows about consumer electronics, and that output is a guess wearing the same clothes as an answer. This is why the coverage vector, rather than the answer text, is the measurement.

Push it further to remove the "bigger k would have worked" objection empirically rather than by argument. Rerank the top 200, then top 1,000, and record the rank at which each linking fact first appears: on the corpus from chapter 1, the sole-source clause first appears at rank 3,891 and the BOM row at 5,602, and 4,000 chunks is roughly two million tokens. Reporting the rank turns an assertion into a measured quantity, and it is the same number that tells you the graph is doing something structural rather than something the retriever could learn.

The belief being tested there is that a larger k is a defence, and it is tempting for a good reason: on ordinary questions it genuinely is one, since raising k almost always drags a missing passage into the context eventually. What breaks it here is that the quantity a retrieval team watches — how well the top-ranked chunks match the question — stays healthy while the property that decides the answer varies underneath it. The top hit is squarely about the fire. Question-to-chunk similarity is doing its job. Meanwhile per-fact coverage, which is the quantity that actually predicts a supported answer, reads two-thirds empty. A metric that holds steady while the dangerous property moves is not a defence; it is a blindfold with a number printed on it, and the rank measurement is what removes it.

One question demonstrated this way is an existence proof. The term is borrowed from mathematics on purpose, and the borrowing is exact in one respect and misleading in another. It is exact in that a single case settles the question of whether at all: one question the graph answers and flat retrieval structurally cannot is enough to establish that the class is non-empty. It misleads if anyone reads it as a measurement, because it says nothing about how often that happens in production, which is the question the funding decision actually turns on.

What an evaluation has to hold fixed#

A staff engineer proposes replacing flat RAG with GraphRAG over an 80,000-document corpus of runbooks, design documents and incident reports; the current system scores 72% on the team's answer-quality eval. Before approving the work, three things are worth seeing: the shape of the query log, one worked demonstration of the kind above on a question from that log, and a cost projection built the way chapter 6 built one, with churn in it. Runbooks and design documents are revised constantly, so the refresh line will not be small.

The evaluation itself is cheaper than people expect, and its design decides whether the result means anything. It runs over a held-out set, meaning questions kept aside from development so that no prompt, schema, threshold or index in either system was tuned while anyone could see them, which is what makes the score an estimate of behaviour on questions the system has not met.

Draw the questions from the query log, not from anyone's imagination, and specifically not from the person proposing the graph. A hand-written set will over-represent the multi-hop shape by a wide margin, because those are the questions the author finds interesting; the query log's actual distribution might be 8% multi-hop, and 8% is the ceiling on the gain any traversal can produce.

That ceiling is worth deriving rather than asserting, because it is the fact that makes the whole exercise worth doing. A traversal changes the answer only on questions that need one, and on every other question the two arms are handed contexts that differ in packaging and not in content. So the largest possible aggregate improvement is the share of traffic with the multi-hop shape, multiplied by how much of that share the graph converts — and since the second factor is at most all of it, the share alone is the ceiling. Any evaluation reporting a gain larger than the multi-hop share of its question set has either sampled a different distribution from production or is measuring something other than retrieval.

Stratify the sample by question shape before you run anything, which means fixing a quota of questions per shape in advance and sampling within each, so the result can be read per stratum. Two hundred questions split across four or five shapes gives 30 to 50 per stratum, which is enough to see a large effect within a stratum and not enough to see a small one, and knowing that in advance is better than discovering it during the argument afterwards.

Score whether the cited evidence supports the answer, not whether the answer reads as correct. That single choice removes the most common way these evaluations mislead. A generator writing from its parametric knowledge — what the model already carries in its weights from pre-training, rather than anything it was handed at query time — can produce the right answer with citations that establish nothing, and a system that does that scores well and fails the moment the question moves off the well-known part of the distribution. The grader is an LLM judge, which is a model prompted to score another system's output against a rubric and calibrated here against a human-scored subset of 40 questions; it is shown the question, the answer, and the cited evidence only, and asked whether the evidence entails the answer, meaning that the answer follows from what is cited without needing anything the judge knows independently.

Withholding everything but the question, the answer and the cited evidence from the judge is what makes the score measure retrieval. A judge that can see more than the system saw is scoring the union of the system's evidence and its own knowledge, and the two arms differ only in evidence, so the extra knowledge lands on both sides and dilutes exactly the difference the experiment exists to detect. The narrower the judge's view, the sharper the contrast between arms.

Hold the budget, the generator and the prompt fixed across both arms. Only the retrieval differs.

def score_pair(q, graph_sys, flat_sys, judge):
    g_ctx = graph_sys.context(q.text)
    f_ctx = flat_sys.context(q.text, budget=count_tokens(g_ctx))
    rows = {}
    for name, ctx in (("graph", g_ctx), ("flat", f_ctx)):
        ans = generate(q.text, ctx)                       # identical generator both arms
        rows[name] = {
            "supported":  judge.evidence_entails(q.text, ans, ctx),
            "abstained":  ans.abstained,
            "latency_ms": ctx.latency_ms + ans.latency_ms,
            "tokens":     count_tokens(ctx),
        }
    return rows

Here is what a naive version of this measures instead, and what each error does to the number.

Read that table by its third column rather than its first. Six of the seven errors inflate a score, and five of those six inflate the graph either alone or alongside flat. Only one of them, the missing latency and cost axis, does something else. A set of mistakes that almost all push the same way is not a list of independent risks — it means the expected value of a carelessly run evaluation is a number too high, and by an amount nobody can bound. That asymmetry exists because each naive choice removes a constraint, and every removed constraint gives the more elaborate system more room to look good in.

Four points, and where they came from#

The result comes back at 76% for the graph against 72% for flat, at roughly nine times the ingestion cost. The aggregate alone does not decide anything, and the first move is to stop looking at it.

Two hundred questions, four points, is eight questions. A paired comparison is the right lens: on this run the graph was supported and flat was not on 19 questions, flat was supported and the graph was not on 11, and the remaining 170 agreed. A McNemar test, which asks only whether those two disagreement counts are far enough apart to be more than chance and ignores the questions both arms got the same, gives a p-value around 0.20 on 19 against 11, so the aggregate difference is not distinguishable from noise at n=200. That is a fact about the sample size rather than about the graph, and it is worth stating plainly to whoever is reading the result, because a 4-point headline invites a decision the data does not support.

Those three counts are worth checking before trusting anything built on them, and they close in two independent ways. First, 19 and 11 and 170 sum to exactly 200, with no residual — the three buckets partition the question set, so no question is sitting in an unclassified fourth bucket where a disagreement could hide. Second, 19 minus 11 is 8, which is exactly the eight questions the four-point headline is made of. That second identity is the one that changes how the result reads. The 170 questions both arms answered the same way contribute precisely nothing to the difference between the systems, however they were answered, so the entire aggregate gap is carried by thirty questions, and within those thirty by a net of eight.

That is also what the p-value is about, and it can be reached two ways that share no working. Treat the thirty disagreements as thirty coin flips and ask how often chance alone splits them at least as lopsidedly as 19 against 11: summing that tail on both sides gives a probability of about a fifth. Separately, the continuity-corrected chi-square form of the McNemar test, which never enumerates a tail and instead squares the gap between the two counts and divides by their total, lands on the same fifth. Two instruments with no step in common agreeing to that precision means the figure is a property of the counts rather than of either method. The residual worth naming is that the uncorrected form of the same test gives a noticeably smaller probability, so 0.20 is the conservative reading, and which correction was used is the kind of thing to state rather than leave for someone to rediscover.

The 11 questions where flat won and the graph lost are also information. Traversal answers can be worse on single-hop questions, because the graph hands the generator triples and snippets where the flat arm handed it whole passages, and losing the surrounding prose costs accuracy on questions whose answer is a paragraph rather than a relation.

The tempting reading of a four-point aggregate is that it is a small version of a large win — that the graph is a bit better everywhere, and more of it would be better still. An aggregate cannot tell you that, and the reason is structural rather than statistical. The aggregate is an average weighted by how many questions of each shape happen to be in the set, so a large gain on a small stratum and a small gain spread evenly produce the same number. It is a quantity that stays fixed while the thing the build decision depends on — where the gain sits — varies freely underneath it. Two teams can report the identical four points and need opposite architectures.

Sizing the set for the effect you expect#

Since the four-point result turned out to be undecidable at n=200, it is worth doing the arithmetic before running rather than after. What a paired comparison needs is enough discordant pairs — questions where the two arms disagree — and the discordant rate is usually 10% to 20% of the set. To call a 4-point aggregate difference at conventional confidence you need roughly 700 to 900 questions, which is a scoring bill people do not want. To call a 30-point difference within one stratum you need about 35, which you can afford easily.

The gulf between those two set sizes is not obvious and is not a matter of one effect being eight times the other. Two quantities move at once as an effect grows. A larger true difference produces a larger gap between the two disagreement counts, and it also produces more disagreements in total, since the arms diverge more often. The gap has to outrun the ordinary spread of a count, and that spread grows only with the square root of the number of disagreements. So a difference that doubles buys much more than double the detectability, and a difference that halves costs much more than double the questions. That is why 30 points needs a few dozen questions and 4 points needs several hundred.

That asymmetry is the practical guidance. Do not build a 200-question set hoping to detect a small aggregate gain, because you cannot. Build a 200-question set stratified so that the stratum where you expect a large effect has 35 to 50 questions in it, and read the result per stratum. The aggregate then becomes a summary statistic you report for context rather than the number the decision hangs on, and the study is powered for the finding that would actually change what you build.

Shipping the 27-point POC#

The other case from chapter 6 — 41% to 68% on a frozen snapshot, with a production run rate near $55,000 a month — is a different decision because the effect is far too large to be noise. The gap between those two figures is where the name comes from, and at that size no plausible sample of a few hundred questions produces it by chance. Whether to ship still depends on things the POC did not measure, and the honest answer to "would you ship it" is: not as proposed, and yes in a scoped form once three of them are known.

The refresh behaviour is the first, and chapter 6 gave the cheap version: re-extract a 500-document sample after a week of real edits and observe the propagation, the community staleness rate and the contradiction rate. The second is the question distribution, since a 27-point gain measured on an eval set built for multi-hop questions tells you nothing about the share of production traffic with that shape; running the same eval questions through the query-log classifier gives the weighting. The third is citation support rather than answer accuracy, because a POC scored on accuracy alone cannot distinguish a graph that found the evidence from a generator that already knew the answer.

Those three are the same three failures the naive-choices table listed, met in a case where the effect is real. A large effect is protection against noise and against nothing else. The question distribution can still be wrong, the scoring can still be measuring the model's memory, and the refresh cost can still arrive after the decision — and none of those is made smaller by the gap being 27 points rather than four.

With those three, the shape of a shippable version is usually clear: graph over the document classes that carry relations, router in front, flat pipeline retained and still serving most traffic, per-stratum quality tracked as a standing eval rather than a one-off, and a monthly extraction budget that somebody has agreed to.

What carries forward into running it is the pair of habits this chapter is built on. Every answer exposes its traversal with a document behind each edge, which is what makes an error legible instead of merely wrong. And the graph's value is re-measured against the flat baseline on the same held-out questions each time the corpus, the extractor or the schema changes, because all three of those move, and the 32-point stratum that justified the build is the number you are entitled to lose without noticing.