Independent opinions have no hop to traverse

A planning document lands in review proposing that the consumer-electronics review search be rebuilt on GraphRAG in the next quarter. The corpus is 4.1 million product reviews averaging 62 words each, covering about 9,000 models across eleven categories. The rationale in the document is that the current system "only does similarity" and that a knowledge graph would let it "reason across the corpus".

Three terms in that proposal decide the answer, and they are worth fixing before any argument leans on them. A knowledge graph is a set of documents rewritten as things and the connections between them: each entity the documents mention becomes a node, and each relationship a document states between two entities becomes an edge. A hop is one edge walked, and a traversal is a chain of hops. GraphRAG — graph-based retrieval-augmented generation — answers a question by starting at the entities the question names, walking edges outward, and gathering the documents behind them, rather than scoring every chunk against the question directly. Whether that machinery repays its cost is decided by a single property of the pairing between a set of documents and the questions asked of it: whether answering a question requires a fact taken from one document in order to make a second document findable at all.

What the query log says the questions are#

The query log settles the question faster than any architecture discussion. Sampling 5,000 real queries and clustering them by shape gives a distribution that is not close:

The five shares add to a hundred with nothing left over, which is worth noticing before reading anything else off the table. It means the last row is a genuine remainder rather than an unmeasured tail, so any question shape the clustering missed has to be hiding inside one of the four named rows rather than beyond them. That is what makes the rest of the reading safe.

Read the fourth column rather than the second. Every one of those shapes is answered by evidence that sits inside individual reviews, gathered across many of them. Nothing in the list requires a fact from review A to make review B retrievable, which was the property from chapter 1 — where a news report named a factory's tenants, a supplier contract named a part, and a bill of materials named a product line, and no chunk resembled the question — that made the Shenzhen question unanswerable at any k. A single review of the X2 that says the battery lasted two days and then degraded is a complete unit of evidence on its own, and 340 more of them are 340 more complete units. The answer is an aggregation over independent facts, and aggregation is not traversal.

The distinction is worth stating in the two words a reader already has. An aggregation asks the same question of many documents and combines the answers; the documents do not need to know about each other, and they can be collected in any order or all at once. A traversal asks a question of one document in order to learn what to ask of the next; the order is forced, because the second query cannot be written until the first has returned. Every row in that table is the first kind. A graph is machinery for the second.

What the graph would actually contain#

Work out what extraction would emit here, because the shape of it is the argument. A reasonable schema for reviews is Review — mentions → Aspect, Review — about → Product, Aspect — sentiment → a polarity value, which here is positive, negative or neutral. Run that over 4.1 million reviews and you get a star: millions of leaf review nodes, each connected to one product and a handful of aspects, with no edges between reviews at all. The graph has a diameter of about four — the diameter being the longest of the shortest paths between any two nodes, so a small one means there is very little structure to walk — and there is no path from any review to any other review except through the product node they share.

That figure of four is not an estimate; it falls straight out of the schema, and following it out is the quickest way to see how little there is to walk. Two reviews of the same product are two hops apart: up the about edge to the product, and back down the about edge of the other review. An aspect hanging off the first review and an aspect hanging off the second add one hop at each end, which gives four, and there is nothing longer in the component because every node is either a leaf review, a leaf aspect or a product hub.

The diameter of about four and the absence of any route between reviews except the product together pin down a design decision the schema does not announce. If Aspect nodes were shared types — one battery life node that every review in the corpus pointed at — then reviews of different products would be two hops apart through it, and the claim that the product is the only route between reviews would be false. So the aspects here are per-review instances, and the consequence is that the picture is not one star but about nine thousand disconnected ones, one per model, with no edge anywhere that crosses from a model to any other model. The word star is doing analogy work and it breaks precisely there: a star has a centre that everything reaches, and this graph has nine thousand centres that reach nothing but their own leaves. The comparison row in the query log — is the X2 or the X3 quieter, at 18% of traffic — therefore gets no traversal even in principle. It is two independent lookups in the graph, exactly as it is two independent lookups in a flat index.

That structure already exists in your database as two columns.

# The "traversal" the graph would perform, expressed against what you have today.
hits = index.search(
    query="battery life",
    filter={"product_id": "vellum-x2", "review_date": {"gte": "2024-01-01"}},
    top_k=200,
)
summary = summarise_map_reduce(hits, question="what do people say about battery life?")

The filter is the about edge. The similarity search is the mentions edge. The map-reduce — summarise each retrieved review on its own, then summarise those partial summaries into one answer — is what a community summary would have been, except that it runs over exactly the reviews this question needs rather than over a clustering computed months ago, so it reflects yesterday's reviews and does not go stale. A product ID column in a metadata filter is a materialised one-hop join, and the reason it feels like less machinery than a graph is that it is less machinery, doing the same work for this shape of question.

The two spellings of the mentions edge are not equally good, and the difference runs against the graph. A node is an identity: a review either points at the battery life node or it does not. A similarity score is graded: a review that says the charge does not hold is close to the query vector for battery life without ever using either word. So the same vocabulary spread that fragments an aspect into several nodes is absorbed by the embedding, because the embedding was never asked to decide whether two phrasings are the same thing. Materialising the edge is what forces that decision, and forcing it is what creates something to get wrong.

Anyone who has written GROUP BY product_id, aspect over a table recognises the filter-then-group shape of that search call. Graph traversal earns its cost when the join path is unknown at query time, variable in length, and not expressible as a column; when the path is always the same single hop and the hop is a foreign key you already store, the graph is a slower spelling of an index lookup.

The bill for building it anyway#

Extraction is per document regardless of how little each document yields. That sentence is the whole cost model, and the arithmetic under it has no free parameters. The bill is the number of documents multiplied by the price of one call, and one call is priced in two pieces: what goes in and what comes out. At 4.1 million reviews, a 400-token system prompt carrying the schema, and 80 tokens of review text, the extraction pass is roughly 2.0 billion input tokens and, at 150 output tokens per call, another 0.6 billion output. At $3 per million input and $15 per million output that is about $15,000 for one sweep, before retries, before the community summarisation pass, and before storage for a graph with something over 20 million nodes.

Every step there is checkable. The prompt and the review together are four hundred and eighty tokens, and four point one million of those is the two billion figure. The same review count at a hundred and fifty tokens out is the six hundred million figure. Priced at the two rates, the input side comes to a little under six thousand dollars and the output side to a little over nine thousand, and those two add to the fifteen thousand with nothing unaccounted for. The identity closing exactly is what licenses using the number: it is not an impression of scale, it is a product of six quantities, and changing any one of them — a shorter schema prompt, a cheaper model, half the corpus — moves it by a factor you can compute rather than guess. What it does not license is treating fifteen thousand as the cost of the project, because the sentence above names three excluded lines by hand, and the next chapter prices them.

The split between the two halves is the part worth carrying away. Fewer than a third as many tokens leave each call as enter it, and yet the output side is about three-fifths of the bill, because output is priced five times higher than input and five beats a little over three. Anyone tuning this cost reaches first for the prompt, since the prompt is the part they wrote; the schema they are trimming sits on the cheaper side of the ledger, and the expensive side is the length of the structured object they asked the model to emit.

The node count follows from the schema in the same way. Each review is one node, each carries a handful of aspect instances, and the nine thousand products plus the polarity values are a rounding error beside millions of leaves. So the total is the review count multiplied by one plus the aspects per review, and at around four aspect mentions each that clears twenty million. The free parameter is the aspect count, and it is the only thing standing between the review count and the graph's size.

What that $15,000 buys, measured on the eval set, is nothing, because the retrieval step is identical. The reviews the graph surfaces for "battery life on the X2" are the reviews the filter surfaces, and the generator writes the same answer from the same text.

What it costs beyond the money is every failure mode from the last two chapters, inherited in full and now attached to a system that was working. Aspect nodes fragment exactly the way company names did: battery, battery life, batt, power, charge holding arrive as separate nodes, and a traversal from one of them sees a slice of the reviews, silently, in precisely the pattern chapter 3 described. Provenance is worse here than in the supplier corpus, because reviews contain sarcasm, quoted complaints from other people, and comparisons — "unlike my old phone, this battery is fine" — and an extractor that emits a negative sentiment edge from that sentence has produced an edge its source document does not support, with a real quote attached. The audit and verification machinery from chapter 4 then has to run over 20 million edges instead of 1.1 million.

The properties that make traversal worth it#

Rather than arguing case by case, it is worth writing down what the supplier corpus in chapters 1 and 2 had that the review corpus does not. Five properties, and the review corpus fails four of them.

The table is read across, not down. Neither column is a score, and a corpus does not qualify by collecting enough yeses; the useful content is in the five gaps between the two cells of each row, and each gap names a different thing that would have to become true before a traversal did any work. The second row is the mildest of them — products do recur with stable identity, which is why the review corpus gets a "weakly" rather than a "no" — and it is also the row that costs nothing, because the stable identity already exists as a column. A corpus that failed only that row would be an entity-resolution problem. This one fails the first and the fourth, which are the two that decide whether there is a hop at all.

The third row deserves a note, because it is the one people skip. Extraction over documents that state relations — contracts, org charts, incident reports, citations, code imports, changelogs — has a factual answer that a human could check by reading a sentence. Extraction over documents from which relations must be inferred is a judgement call in every row, and the graph then contains a model's opinions with the same visual authority as a contract clause. A graph whose edges are inferred sentiment is a materialised classifier output, and it inherits every error of that classifier permanently, whereas classifying at query time over the retrieved reviews at least lets you change your mind.

Routing by where the evidence sits#

The architecture most real systems end up with follows from the table: two indexes and a router, where the router's job is to classify a question by whether its evidence spans documents.

def route(question: str) -> str:
    plan = classify(question)   # small model or a trained classifier over the query log
    if plan.needs_linking_fact:            # entities not co-mentioned in any document
        return "graph"
    if plan.is_aggregate_over_one_entity:  # 61% of this log
        return "flat+filter"
    return "flat"

A router built from the query log above sends 91% of traffic to the flat path on the first three rows of the distribution alone, which reframes what the graph is being justified against. It is not competing with the flat system for the whole product; it is competing for whatever slice of traffic actually needs it, and the ingestion cost has to be justified by that slice.

The word "alone" in that sentence is doing real work, and the code shows why. The fourth row, at 7%, does not reach the graph either: a specific complaint lookup needs a handful of reviews of one model, which is single-document evidence gathered a few times over, so it falls past both branches and returns flat. Add it and the flat paths together take almost the whole log. The graph's share is not the nine points left over after 91; it is only the traffic where the first branch fires, and how much traffic that is has not been measured yet by anything on this page. The query-shape clustering cannot answer it, because a question's shape and the span of its evidence are two different attributes and the clustering only recorded the first.

Testing the proposal before building anything#

Where a review corpus does have hops#

The argument is about the pairing of corpus and questions, not about the domain, and it is easy to construct a review question with real hops. "Which models use the battery cell that reviewers complain about swelling?" is a genuine multi-hop question: reviews mention swelling for model A, a teardown document names the cell in model A, and a supplier document lists which other models use that cell. The evidence spans three documents and no chunk resembles the query, which is chapter 1's structure exactly.

The right build for that is not a graph over the reviews. It is a graph over the teardowns and supplier documents, roughly 40,000 documents rather than 4.1 million, with the review index left exactly as it is and consulted at the leaves. The traversal finds the affected models; the flat index answers what people say about them. Ingestion cost drops by two orders of magnitude because it is proportional to the number of documents that contain relations, and the review corpus contains none.

That factor of a hundred is exact for the extraction line and only approximate for the bill, and the distinction is worth keeping. Extraction is per document, so cutting the document count by a hundred cuts extraction by a hundred. The passes that follow extraction are not per document — they scale with the graph that comes out — and a small corpus of teardowns and supplier agreements is dense in entities in a way a review corpus is not. The next chapter takes those passes apart; the safe form of the claim here is that the term everybody quotes falls by two orders of magnitude, and the terms nobody quotes fall by less.

The two defences behind a decision to decline#

The tempting defence in all of this is the measurement. A team that has sampled its query log, labelled the evidence spans and computed a ceiling has done real work, and the number it produced feels like protection against building the wrong thing. It is protection against one thing only. A log records the questions users have learned this system can answer, and a system that has never answered a cross-document question does not receive many: people ask it once, get a fluent wrong answer or nothing, and stop. So the multi-hop share is a quantity that can sit perfectly still while the demand it is meant to measure moves, which is the failure shape to watch for in any metric used to decline a build. The repair is to ask a sample of users what they gave up asking, and to label those questions too.

The review corpus survives that objection, and the reason it survives is the useful part. The verdict here rests on two instruments that share no step. One is the log, which counts what people ask. The other is the schema, which says that a review is a complete opinion and that no fact in one review makes another review findable — a property of the documents that holds whatever anybody asks. Both return the same answer, so the decision is safe. On a corpus where the two disagreed, the log would be the weaker of the pair, because the structural property cannot be biased by what users have given up on.

The second defence worth naming is the unit cost. Extraction is priced per document, that price is measurable, and a measured price feels like control. It is control over the wrong variable, because the cost of reading a document barely moves with what the document yields while the value of reading it moves by orders of magnitude. A supplier agreement states relations that a hundred later questions traverse. A sixty-two-word review states one opinion that no question traverses at all. A team that carries a unit cost across from a contract corpus to a review corpus is carrying a number that stays true and a conclusion that does not.

None of this says aggregate questions are easy or that flat retrieval answers them perfectly. Summarising 340 reviews faithfully has its own hard parts — sampling bias when you take top-k by similarity rather than a representative sample, recency weighting, and the tendency of a generator to report the vivid minority complaint as the consensus. Those are real problems and a graph solves none of them, which is the point of raising them: the failure mode in this corpus is aggregation quality, and building traversal machinery is an answer to a different question.

Nor does it say graphs are only for supply chains. Corpora with stated relations and cross-document evidence are common — engineering incident reports referencing services and prior incidents, legal filings referencing parties and precedents, research literature with citations, codebases with call graphs, medical records with medications and conditions. What those share is not their subject matter but the five properties in the table.

The honest position on the review corpus is that the proposal should be declined, and the reason to give is that these are aggregate questions over independent facts, so there is no hop to traverse, and flat retrieval with metadata filtering already does the work. The proposal deserves one concession: if a meaningful share of the query log turned out to have the cross-document shape, the answer would be a scoped graph over the small documentary corpus rather than a rebuild.

Declining is easy when the graph buys nothing. The genuinely hard case is when it buys something, and the argument moves to whether the gain is worth the ongoing cost — which turns out to be a much bigger number than a one-off extraction estimate suggests, for reasons the next chapter takes apart.