A multi-hop question has no chunk that resembles it, at any k

An internal assistant sits over a corpus of 60,000 documents: contracts, supplier agreements, bills of materials, incident reports, policy pages, and four years of procurement email exported to text. On a Tuesday morning somebody in support asks it what the refund window is for EU customers, and it answers correctly in 1.4 seconds, citing the two policy pages that state fourteen days plus the statutory cooling-off period. An hour later somebody in operations asks which of the company's product lines depend on suppliers affected by the Shenzhen factory fire. The assistant answers in 1.6 seconds, names two product lines with the same confident tone, and cites a corporate risk-management policy and a news summary of the fire. One of the two product lines is wrong. The other is right by coincidence, in the sense that nothing the model retrieved actually establishes it.

The interesting part is that the second failure is not a harder version of the first. It is a different kind of failure, and no amount of tuning the retriever that answered question one will make it answer question two.

Nothing about how the two answers arrived says so. Both came back in about a second and a half, two-tenths of a second apart. Both were fluent, both carried citations, and both citations pointed at real documents that genuinely discuss the subject. The quantities an operator watches — latency, whether citations were produced, whether the cited documents are on topic — hold steady across the two questions, while the property that decides whether an answer is worth anything moves from present to absent underneath them. That combination is what makes this failure survive review, and it is worth carrying through the rest of the chapter.

The machinery built for the second kind of question is worth naming before any argument leans on it. A knowledge graph is a corpus rewritten as things and the connections between them. Each company, part, facility and product line mentioned in those 60,000 documents becomes a node. Each relationship that some document states between two of them becomes an edge, and every edge keeps a pointer back to the sentence that asserted it.

The name for a pipeline that builds such a graph ahead of time is GraphRAG, graph-based retrieval-augmented generation. It has a model read each document and report the entities and relations it finds, and then answers a question by starting at the entities the question names and walking edges outward, gathering the documents behind them, instead of scoring every chunk in a corpus against the question the way the retrieval strategies of the previous topic do. Walking a single edge is a hop, following a chain of them is a traversal, and a question that cannot be answered without more than one hop is a multi-hop question, which the Shenzhen question turns out to be.

The problem all of that exists to solve is the one the second answer ran into: evidence spread across documents that do not individually resemble the question. You meet it wherever a system is asked how things inside an organisation connect — which suppliers a fire touches, which services an incident spans, which contracts a policy revision affects — and in the tooling built for that shape of question, Microsoft's open-source graphrag pipeline and the graph-database vendors' retrieval stacks among them. The rest of this topic is about the conditions under which the machinery repays what it costs, and the first of those conditions is visible in the anatomy of the question that failed.

Where the evidence for the second question actually lives#

Write down what a correct answer requires, document by document. A news summary in the corpus says a fire on 3 March destroyed Building C of the Longhua industrial park in Shenzhen, and names the tenants: Hanyu Precision, Kestrel Components, and two others. A supplier agreement, signed in 2022 and stored in the contracts folder, records that Hanyu Precision supplies board-to-board connectors, part number CN-4471, under a sole-source arrangement. A bill of materials for the Aurora handset lists CN-4471 in the mainboard assembly. Three documents, none of which mentions the other two, and the chain runs fire → tenant → part → product line.

The labels on those arrows are the load-bearing part of the picture. Each link is asserted by exactly one document, and no document asserts two consecutive links. The news summary knows about the fire and the tenant and stops there; the contract knows about the tenant and the part and has never heard of the fire; the bill of materials knows about the part and the product line and has never heard of either. A reader can chain them because they can hold three documents in mind at once. A retriever cannot, because it is about to look at each of them separately.

Now look at the query embedding. "Which of our product lines depend on suppliers affected by the Shenzhen factory fire" is lexically and semantically about fires, Shenzhen, suppliers and product lines. The news summary scores well against it. The corporate risk policy, which is full of the words supplier, disruption and dependency, scores well against it too, and contributes nothing. The supplier agreement for CN-4471 is a nine-page legal document about payment terms, quality tolerances and delivery windows that never uses the word fire, never mentions Shenzhen by name, and never mentions Aurora. The bill of materials is a table of part numbers.

Here is what the top of the ranking looks like, with cosine similarity against the query, from a run on this corpus with a standard 1,024-dimension embedding model and 512-token chunks.

[
  {"rank": 1,  "score": 0.681, "doc": "news/2024-03-longhua-fire.md",        "note": "names tenants"},
  {"rank": 2,  "score": 0.643, "doc": "policy/supplier-risk-management.pdf", "note": "generic, no facts"},
  {"rank": 3,  "score": 0.629, "doc": "policy/business-continuity-2023.pdf", "note": "generic"},
  {"rank": 4,  "score": 0.611, "doc": "email/procurement-2024-03-05.txt",    "note": "asks the same question"},
  {"rank": 10, "score": 0.554, "doc": "news/2023-11-tianjin-port-delay.md",  "note": "different incident"},
  {"rank": 100,"score": 0.478, "doc": "policy/sole-source-approvals.pdf",    "note": "generic"},
  {"rank": 3891,"score": 0.312,"doc": "contracts/hanyu-precision-2022.pdf",  "note": "REQUIRED"},
  {"rank": 5602,"score": 0.288,"doc": "bom/aurora-mainboard-rev-c.csv",      "note": "REQUIRED"}
]

Read that block as a function rather than as a list, because the function is what settles the argument. Flat retrieval computes a chunk's score from two things and nothing else: that chunk's vector and the query's vector. The size of the result set appears nowhere in the computation. So every chunk's score is already fixed before the retriever has been told how many chunks to return, and so is every chunk's position in the sorted list. Raising k cannot move a chunk up or down. It can only move the line below which the sorted list is thrown away.

That is what makes the last two entries hopeless rather than unlucky. The score of the Hanyu contract against this question was settled by two vectors, and neither of them knows that a document a few thousand places above it names Hanyu as a tenant of the building that burned.

The two highest-scoring documents that carry no facts explain the shape of the rest of the list, so each is worth a sentence. A policy page about supplier risk is written in the vocabulary of the question — supplier, disruption, dependency — and similarity rewards exactly that overlap, so a document about the topic in general lands close to a question about the topic in particular while asserting nothing that could answer it. The procurement email at rank 4 goes one step further, because it asks the same question: a question is the nearest neighbour of a question, which is why the head of a list like this fills with restatements of what was asked rather than with answers to it.

The instructive readings further down are differences rather than values. The generic risk policy at rank 2 sits under four hundredths of a point below the news summary at rank 1, so the gap between a document that names the tenants and a document that names nobody is narrower than a rounding. The drop from rank 4 to rank 100 is about an eighth of a point across ninety-six places. The drop from rank 100 to rank 3,891 is about a sixth of a point across nearly four thousand places. Similarity falls steeply at the head of the list and then almost stops falling.

A score threshold is the defence most readers reach for at this point, and the flatness is what defeats it. Between rank 100 and rank 5,602 the similarity moves by less than a fifth of a point, while the property that decides the answer — whether a chunk carries one of the three links in the chain — goes from absent to essential inside that same band. A quantity that barely moves cannot be used to select on a property that moves completely, and no choice of cut-off in that band separates the two required documents from the several thousand irrelevant ones they are wedged among.

The ranks settle one more thing on their own. A corpus of 60,000 documents produces at least 60,000 chunks, since no document yields fewer than one, so rank 5,602 sits inside the top tenth of everything stored. The two documents that carry the answer sit at ranks 3,891 and 5,602, which is near the front of this index and still thousands of places below any k a system would actually run. That is not a ranking bug to be fixed with a better reranker; the similarity function is behaving correctly, because those documents genuinely are not about the question as asked. Their relevance is conditional on a fact — that Hanyu Precision was in Building C — which is contained in a different document and is not available to the retriever at the moment it scores them.

This is the structural property worth naming, and it is the answer to the interview question about what makes Q2 hard. The evidence for Q2 is not low-ranked, it is unrankable by the query, because the surface that would make it match is contributed by another document. Flat top-k retrieval scores every chunk independently against a fixed query vector, which is exactly the right design when the answer is a chunk that resembles the question, and is structurally unable to help when it is not.

Contrast the refund question, which has the opposite shape. "What is our refund window for EU customers" is answered by a policy page containing the words refund, window, EU and customers, in a passage written to be exactly that answer. Query and evidence share a surface, one chunk suffices, and the retriever's independence assumption costs nothing. Most of the traffic on most corpora looks like this, which is why flat retrieval is not a stepping stone to be outgrown.

Why a bigger k is the tempting wrong answer#

The instinctive fix is to retrieve more. If the evidence is at rank 3,891, retrieve 4,000 chunks. The arithmetic settles it quickly, and it is worth doing in general form first, because the general form is what lets you check a row you care about rather than a row somebody happened to write down. Context spent on retrieval is the chunk size multiplied by k, since the retriever returns k chunks and each one is capped at the chunk size. At 512 tokens per chunk, k=100 is roughly 51,000 tokens of context, k=4,000 is around 2.05 million tokens, which is beyond the window of every model in production use and would cost, at $3 per million input tokens, about $6 per query for the input alone. Retrieval latency at that k is also not the p95 anyone has budgeted for.

That is the cost objection, and it is not even the strongest one. Suppose you could afford it. At k=4,000 you have handed the generator 4,000 chunks, of which two matter, and asked it to notice that a part number in a CSV appears in a contract whose counterparty appears in a news article. Answer quality against distractor-heavy contexts is a curve that peaks and then declines, as the k sweep in Retrieval strategies for RAG showed on a much smaller scale, and there is no reason to expect the peak to have moved out to four thousand.

Every cell in the tokens column there is that same product, printed to two or three figures: 512 tokens times a k of ten for the first row, times a hundred for the second, times four thousand for the third. Nothing else is in those cells, which is what licenses extending the table — pick a chunk size and a k of your own and the answer is one multiplication. Two residuals come with that licence. The products are rounded before printing, so the ratio between two printed cells is exact only to the rounding, and the column counts retrieved chunks alone: the question, the instructions and any per-chunk header a prompt builder adds are outside it, which makes each figure a floor on what gets billed rather than the billed amount.

The rows are worth reading against each other rather than downwards. The first three differ in one thing only, which is where the ranked list gets cut, and the second column reads no across all three of them: multiplying the input by roughly four hundred between the first row and the third changes the answer to the question that column asks not at all. The fourth row leaves the cut alone and changes the query instead. It costs about a sixth more input than the cheapest row that fails, and a fraction of a percent of the row that technically succeeds, so the price of the fix is not what makes anybody hesitate over it. The last column is: you must know how many hops, or iterate.

The last row is the one that works, and stating it that way — a second retrieval whose query is built from the result of the first — is more useful than naming a technology, because two quite different technologies implement it.

Query decomposition supplies the same second retrieval#

A teammate who says decomposition would answer this without a graph is right about the mechanism and is making a claim that has to be checked case by case rather than dismissed. An agentic loop can do exactly the work described above:

# hop 1
q1 = "which companies had facilities in the Shenzhen Longhua fire?"
# → ["Hanyu Precision", "Kestrel Components", ...]

# hop 2, query built from hop 1's answer
q2 = "what parts does Hanyu Precision supply to us?"
# → ["CN-4471 board-to-board connector, sole source"]

# hop 3, query built from hop 2's answer
q3 = "which product lines use part CN-4471?"
# → ["Aurora handset", "Aurora Lite"]

Each hop is a normal flat retrieval, and what moved between hop one and hop three is the query rather than the depth of the cut. The Hanyu contract that scored 0.312 against the original question scores well against q2, because q2 names Hanyu Precision and asks about parts, which is the surface that document actually has. The score is a function of a chunk and a query; the cut point is not one of its arguments, and the query is the only argument anybody gets to change from outside. That is why the fix takes the form of a different retrieval rather than a larger one.

Nothing about it needs a graph, and on this particular question it works, at a cost of three retrieval round trips and three or four generation calls, which in practice lands somewhere between 4 and 9 seconds end to end depending on the model. If your corpus produces a handful of such questions a day, that is a completely reasonable system, and it costs you no ingestion machinery at all.

That range and the table's latency column arrive at the same place by different routes, which is a useful check on both. The table charges +1.5 s per hop, and this chain is three hops, so the retrievals alone are four and a half seconds — the bottom of the 4 to 9 second band. The rest of the band is the three or four generation calls sitting between the retrievals. So the floor of the range is a property of how long the chain is, and the width of the range is a property of which model is writing the intermediate answers. The hops cannot be overlapped to shrink it, because each query is built from the previous answer and does not exist until that answer does.

When a graph earns its place#

Decomposition and traversal both supply the second retrieval, so choosing between them is a question about conditions rather than about capability. The graph earns its place under conditions you can state in advance, and it is worth being specific rather than gesturing at "complex questions".

The number of hops is not known before the traversal starts. The Shenzhen question happened to be three hops. "Which of our customers are exposed to this fire" might be four, because it runs on through product lines to accounts, and "which regulatory filings will need amending" might be six. A decomposition loop has to decide at each step whether it is done, and that decision is made by a model that cannot see the graph and is therefore guessing; the failure mode is stopping early with a plausible partial answer. A traversal under a hop budget explores the frontier exhaustively and reports what it found at each depth.

The intermediate entities cannot be named from the question. Decomposition works well when the sub-questions are guessable — "compare the revenue of A and B" decomposes cleanly because A and B are in the question. It degrades when the second query depends on an entity discovered in the first, especially when there are eleven such entities and the loop has to fan out over all of them. At a fan-out of ten per hop and three hops, the iterative approach is doing on the order of a thousand retrievals; the graph is doing a two-hop neighbourhood expansion in a few milliseconds because the edges are already materialised.

That thousand is worth deriving, because the shape of it is the argument. A loop that fans out to some number of entities at each hop multiplies its frontier by that number every time, so ten per hop over three hops reaches a thousand entities on the last frontier. A loop that does not know when to stop has to issue a retrieval at every entity it reaches, the last thousand included, since retrieving there is the only way to discover that nothing follows. Adding the hops gives the thousand plus the hundred plus the ten plus the one you started with, so everything above the last hop contributes about a ninth on top of it. That is what licenses quoting the frontier size as the cost and ignoring the rest, and it is also why the count is an exponent in the number of hops rather than a multiple of it.

The same traversal is amortised across many queries. This is the argument that decides most real cases. Building the edges costs money once (chapter 6 puts numbers on it), and reading them costs almost nothing. If the operations team asks fifty variants of "what is exposed to X" every week, you are paying for the fan-out fifty times a week under decomposition and once under a graph. If they ask it twice a quarter, you are not.

Single-hop questions, and answers a corpus does not contain#

A graph does nothing for the refund-window question, and it is worth being clear that the assistant's first answer was not lucky. Single-hop factual lookup over a corpus where the answer and the question share a surface is what dense retrieval is good at, and adding a traversal layer over it adds latency and failure modes with no upside. The same goes for the aggregate questions in chapter 5.

A graph also does not fix a corpus that does not contain the answer. If the bill of materials for Aurora was never ingested, the edge from CN-4471 to Aurora does not exist, and the traversal will return the two hops it can find and stop, which reads exactly like a complete answer unless something in the output distinguishes "no further edges" from "no further edges within this budget". That distinction matters enough that it shows up repeatedly through the rest of this topic.

And it does not, by itself, make an answer verifiable. Notice what went wrong in the opening scene: the assistant produced a fluent sentence naming two product lines, and the reason nobody caught it for a week is that the citations it offered were real documents that supported nothing. A graph answer that shows only its final sentence is no better. The thing to demand, from the very first held-out question you test, is the traversal itself — the intermediate nodes it passed through, and for each edge, the document that asserted it. If the answer names Aurora, you want to see Hanyu Precision and CN-4471 in the trace, and you want the contract and the BOM attached to the edges that carry them.

What the arithmetic leaves behind is a rule about where to spend effort. The size of the result set is outside the scoring function, so tuning it tunes a cut point, and no cut point reaches evidence whose relevance was contributed by a document the retriever never saw while scoring. A second retrieval reaches it, because a second retrieval changes the query. Whether to build that as a traversal or as a decomposition loop is decided by the three conditions above — unknown depth, unnameable intermediates, repeated use — and not by how hard the question feels to a person reading it.

Which raises the obvious question of where those edges come from, what a "node" is when your corpus is prose rather than a database, and how a question in English gets turned into a starting point for a traversal at all. That machinery is the subject of the next chapter, and every failure mode in the four chapters after it is a consequence of one of its parts.