The visual-only arm is a control, not a shippable candidate

A control arm is a configuration you run in order to read the other configurations, and its own performance is beside the point. It removes one contributor from a comparison so that a difference between the remaining arms can be attributed to that contributor rather than to something else that moved at the same time.

The occasion for one here is a proposal on the table: a visual index over 400,000 pages of financial filings. The build is about three weeks of engineering plus roughly 950 GPU-hours to render and encode, the storage grows by 2.1 TB, and the ongoing cost per query goes up by a factor the team has not yet computed. The existing text-only system answers 68% of the evaluation set correctly. Someone senior asks the only question that matters: what does the visual index buy, and how would we know?

The answer is a three-arm comparison, and the arm that makes it informative is the one nobody would ever put in production.

Three arms, one question set, and what is held fixed#

Every arm runs the same 400 well-posed questions from chapter 5 — the adversarial set built so that a page which merely resembles the answer scores differently from a page that contains it — through the same generator, with the same prompt template, the same slot budget, and the same instructions. The only thing that varies is which retrievers feed the candidate pool.

arms:
  text_only:
    retrievers: [bm25, dense_text]
    fusion: rrf
    slots: {text: 8, image: 0}
  visual_only:
    retrievers: [page_image, figure_crop]
    fusion: rrf
    slots: {text: 0, image: 5}
  hybrid:
    retrievers: [bm25, dense_text, page_image, figure_crop]
    fusion: rrf + cross_modal_rerank
    slots: {text: 5, image: 4}

fixed_across_arms:
  generator: {model: claude-opus-5, max_tokens: 1024, temperature: default}
  render: {long_edge_px: 2048}
  prompt_template: v7
  question_set: filings_wellposed_v3   # 400 questions, hard distractors

Three keys there need naming. rrf is reciprocal rank fusion, which merges ranked lists by scoring each item on the ranks it achieved rather than on the raw scores it was given, so lists with incomparable score scales can be combined. cross_modal_rerank re-scores the merged candidates with a model that reads both text and images. slots is the shape of the prompt: how many text units and how many image units the template holds, fixed in advance so a retriever cannot win extra room by returning more candidates.

Text-only is the baseline: what you have, measured on the same set as everything else so the comparison is not against a number from a different evaluation.

Hybrid is the candidate: what you would ship.

Visual-only is the control, and it is there to answer a question the other two cannot. Hybrid minus text-only gives you the total improvement. It does not tell you where the improvement came from, and there are two very different sources. The images might carry evidence the text pipeline never had — the case this whole topic has been arguing exists. Or the visual retriever might be functioning as an ensemble member: a second, differently-biased opinion whose disagreements with the text retrievers, when averaged, cancel some of their errors. Ensembling two mediocre retrievers reliably produces a few points of improvement over either alone, on any corpus, with no visual evidence involved at all.

Those two explanations imply different projects. If the gain is evidential, the follow-on work is resolution budgets, better layout detection, figure-level units — investment in the visual path. If the gain is ensembling, the follow-on work is fusion and reranking, and you have bought an expensive ensemble member that a cheaper second text retriever might have matched. Without the control you cannot distinguish them, and you will spend the next quarter on whichever one the loudest person believes.

The word invites an analogy with the placebo arm of a drug trial, and the analogy is worth stating with its limit attached. A placebo is inert by construction, so its score is a floor and carries no information beyond locating that floor. A visual-only arm is a working retrieval system whose score is a measurement in its own right, which is why the control's own number, and not only the gap it opens, does the work below.

The retriever list is the intended variable, and one thing the YAML does not hold fixed alongside it is worth naming as a design decision rather than a requirement. The arms carry different slot budgets: eight text units for the baseline, five image units for the control, five text and four image for the candidate. Each generator therefore sees a different quantity of evidence, and the hybrid arm sees the most — nine units where the baseline sees eight, four of them page images costing far more context than a text chunk. Some unknown share of the hybrid gain is that larger budget rather than the modality in it, and the confound cannot be cleanly removed, because a page image and a text chunk are not commensurable amounts of context. Read "the same slot budget" above as the same template rather than the same quantity of evidence.

Reading the arm table as gaps#

sufficient@5 is the fraction of questions for which the five retrieved units handed to the generator collectively contain enough to derive the answer, judged against the question rather than against a relevance label. It describes the evidence that arrived; answer correctness describes what the generator did with it. What the three numbers say together is carried in the gaps between the columns rather than in any column's levels.

Sufficiency is a ceiling on correctness, for a mechanical reason. If the five units do not contain enough to derive the answer, a correct answer came from somewhere other than the evidence — a prior, or a lucky guess — which is not a capability anyone is buying. The distance between the columns is therefore what the generator lost after being handed enough. Text-only loses three points of what it was given, hybrid six, and visual-only eight, nearly triple the text arm's loss on evidence the same judgement called sufficient.

Citation grounding is the column that catches a failure the other two miss, and it catches this one by a route with no step in common with the columns above, which is what makes their agreement worth something. It measures the fraction of claims whose cited evidence actually supports them — the crop or chunk the answer points at, judged against the claim. One judgement is made on a retrieved set before generation, the other on a cited region after it, and they share no step. Both rank the arms identically: text-only loses least and grounds best, hybrid sits in the middle, visual-only loses most and grounds worst. Two instruments with no common step landing on one ordering licenses an attribution — the visual arm's deficit is in reading the evidence rather than finding it — and leaves a residual, since neither says how much of that reading loss is the model and how much is the resolution the pixels arrived at.

The visual-only arm loses 13 points of grounding relative to text-only, because a model reading a chart produces a value and a region that plausibly frames the chart without the region proving the value. Hybrid at 0.88 is three points below text-only, which is the price of introducing evidence whose support is harder to verify. That is a number to watch rather than a blocker, and a system whose correctness rises while grounding falls is one whose answers are getting harder to check, which is worth knowing before rather than after a customer disputes one.

Visual-only at 0.41 is the informative number, and it is informative because it is high, not because it is low. It says that images alone, with no text pipeline whatsoever, independently answer 41% of this question set — including a large share of questions the text arm gets wrong. That is a lot of standalone evidential capacity, and it makes "the six points are pure ensembling" hard to sustain: an ensemble member that can answer four questions in ten by itself is contributing evidence, not just decorrelated noise.

The cost column looks wrong on a first pass and is not. Hybrid runs four retrievers and a cross-modal reranker where visual-only runs two retrievers and no reranker, so hybrid should be the expensive one, and it is cheaper. The resolution is in the slot budgets. Visual-only sends five images to the generator and hybrid sends four, and an image at this render size carries enough context to dominate everything else on the bill. Divide the two costs and the ratio comes out close to five-to-four — the ratio of the image slot counts, with the retriever counts playing no part. The residual runs the other way from what the extra retrievers and the reranker predict, hybrid sitting slightly below the pure slot-count prediction where it ought to sit above, and the difference is smaller than the last digit the table quotes. The p50 column repeats the story at the same ratio. The planning rule both license is that the per-query bill tracks how many images reach the prompt, so the lever that moves it is the slot budget rather than the retriever list.

Slicing by where the evidence lives#

The arm-level table still cannot show which questions moved, and that is what makes a fusion change defensible. Label every question in the set with the modality that holds its evidence — determined once, by the annotator, at question-authoring time — and score each arm per slice.

The two tables are views of one measurement, so they have to reconcile. Correctness over the whole set is an average of per-question scores, and the four slices partition the set with no question in two of them and none left out. Multiplying each slice's score by its share and adding the four products therefore has to return the arm's overall number, up to the rounding in the printed figures. Do that down the text-only column and it returns 0.68, the figure in the arm table. The closure licenses something otherwise unchecked: both tables describe the same four hundred questions at the same shares, with no slice silently dropped or counted twice.

Run the same recombination down the other two columns and it does not close. The visual-only column comes back two points below the 0.41 the arm table prints, and the hybrid column four points above the 0.74. Rounding cannot open a gap that size — shares printed to a whole percent against scores printed to two decimals leave a residual well under a point, which is what the text-only column demonstrates. At least one cell in one table does not belong to the other, and the response is to ask for the per-question join behind both rather than to take whichever number supports the preferred decision. Everything below reads each table on its own terms and never multiplies across the two.

Read the second row first: the hybrid gain is concentrated there, going from 0.19 to 0.61 on the slice where the text pipeline structurally cannot see the evidence. That is the direct answer to the ensembling hypothesis, stated per question rather than inferred from an aggregate. How much of the gain it is repays being exact about. Row two moves by more than forty points on close to a quarter of the set, which weighted into the whole already exceeds the six points separating the two arms overall, and rows three and four move upward too. That is the reconciliation failure above seen from the gain side, and the reason to say the gain is concentrated in row two rather than that row two is all of it.

Read the first row next: 0.86 to 0.85 says the change did not regress the majority population. One point on 216 questions is inside the noise band you should have computed alongside it. That band comes from a bootstrap — resampling the question set with replacement a few thousand times to see how far the score wanders — which gives roughly ±0.04 at 95% here, so both the non-regression on row one and the improvement on row two are being read at the right confidence.

The slice table is also the artefact that answers the fusion question from chapter 6 — where weights rather than evidence decided which modality won slots — honestly. Any weight change, reranker swap or slot-budget adjustment gets evaluated on all four rows, and a change that lifts row two while dropping row one below its confidence band is a trade, not an improvement, no matter what the overall number does. Two populations moving in opposite directions is exactly the thing an aggregate is designed to hide.

Row two also shows hybrid at 0.61 against visual-only at 0.79, and that gap is a finding rather than a rounding artefact. On the questions where images hold the evidence, the hybrid system is 18 points worse than a system that only looks at images — because text candidates are still competing for slots and still winning some of them. That is the residue of chapter 6's calibration problem, quantified, and it is the strongest available argument for spending the next sprint on per-query slot allocation rather than on the encoder.

The p95 that comes with the visual arm#

The p95 latency number is a decision, and it is a decision that should be made explicitly rather than absorbed. Four options, with what each costs:

The last row is worth stating because it is often proposed as the fix and is not one. Embedding a corpus is a build-time cost that was never in the query path, and a cost cannot be taken out of a path it was never on, so caching it saves nothing at query time.

That rule and that row disagree, and the disagreement sits in the label rather than in the number. The rule predicts a p95 unchanged at 1,900 ms; the row's own note says the caching was already assumed, predicting the same; the row prints 1,700 ms. The reading that reconciles all three is that the two hundred milliseconds belongs to the other precomputation in the same neighbourhood — pre-rendering pages to disk, which takes a per-candidate render out of the query path and which the paragraph below calls the cheapest latency win available. Under that reading the number is defensible and the row is named after the wrong artefact. Worth correcting in the table; worth reading, meanwhile, as a saving that pre-rendering buys.

The query-time cost is rendering candidate pages to PNG (about 90 ms each if not pre-rendered), reranking across modalities (180 ms), and the generation step processing four images of roughly 3,000 tokens each. Pre-rendering every page to disk at index time removes the render cost and adds 2.1 TB of PNGs to the 400 GB of vectors, which on this corpus is the single cheapest latency win available and is why it belongs in the build rather than the query path.

Gating is the most effective and the most consequential. Routing only visually-flagged queries through the image path brings p95 back near baseline for the majority of traffic, and the 11-point drop on the figure slice is the cost of every misroute being unrecoverable. Whether that trade is right depends on something the evaluation cannot tell you: what a wrong answer costs relative to a slow one. For an internal research assistant, 1.9 s is nothing and gating is a needless risk. For a customer-facing product with a latency budget in the service agreement, gating is the only option on the table, and the right response to the 11 points is to improve the classifier's recall on visual queries by biasing it toward routing — a false positive costs 1.5 s, a false negative costs an answer.

That last recommendation can be derived rather than asserted, starting from an identity inside the gating row: two of its own numbers have to agree. The row costs 0.11 on the figure slice, and that slice is 23% of the set. If the loss falls only there, the overall loss is 0.11 weighted by that share, which comes to 0.03 — the other number on the same row. The column closes with nothing left over, and the closure licenses a claim the row does not make in words. Gating does no quiet damage to the majority slice; every point it costs is a point on a question it misrouted.

The classifier's accuracy is where the same row turns into a trap. Eight milliseconds and roughly 93% accurate reads as a well-built component. But a router that sent every query to the text path without looking at it would already be right on every question whose evidence is not figure-only, which is more than three quarters of this set. Overall accuracy stays high whatever a router does with a small minority class, so it cannot distinguish a router that learned the distinction from one that learned to answer "text". The reported quantity holds steady while the property anyone cares about swings.

Recall on the visual class can distinguish them, and it is recoverable from the two figures already on the row. A misrouted figure-slice query loses the visual arm entirely, falling from hybrid's 0.61 toward text-only's 0.19 — a little over forty points for each one misrouted. The row says the slice as a whole drops by eleven. Eleven points of realised loss against forty-two available per misroute puts roughly a quarter of figure-slice queries outside the image path. A router right about nine times in ten overall is wrong about one time in four on the only class it was built to catch, and both hold at once because that class is small. Hence recall rather than accuracy, and hence biasing toward routing.

Where the slice proportions come from#

The slice proportions are an assumption, not a measurement, and they are doing more work in the overall numbers than anything else in the table. This set is 23% figure-answerable because the annotators built it that way.

What the assumption does to a conclusion can be worked out rather than guessed, and two ways of working it out give different answers. The quick way scales the headline six-point gain by the ratio of the new figure share to the old, which is where "about two points" comes from at an 8% figure share. The honest way reweights every row: multiply each row's arm-to-arm gain by its new share and add the four products. They agree only if the entire gain sits on row two, and the reconciliation check above showed it does not, so the reweighted calculation returns the larger number — a difference that can decide a marginal build. If real traffic is 8% figure-answerable the hybrid arm's overall gain shrinks toward the smaller of the two and the latency cost is much harder to justify; if real traffic is 40% the case is overwhelming under either method, which is the useful thing about that extreme.

Sampling the slice proportions from production query logs — even roughly, even by hand-labelling 200 sampled queries — is a cheaper and more decision-relevant piece of work than another point of encoder quality, and it is routinely skipped because it is not an engineering task.

How wide the interval is on a slice#

Set size interacts with slicing in a way that catches teams out. Four hundred questions is comfortable for an arm-level comparison and thin once you cut it four ways: the figure slice here is 92 questions, so its confidence interval is roughly ±0.10, and the synthesis slice at 16 questions is decorative — a single question moving shifts it by six points.

Both figures follow from one relation worth carrying rather than looking up. The width of a proportion's interval falls with the square root of the number of questions behind it, so cutting a set to a quarter of its size comes close to doubling the interval — which is how a whole-set band of ±0.04 becomes a figure-slice band near ±0.10. Run the relation backwards and it prices any target. Halving an interval costs four times the questions, so pulling the figure slice down to ±0.05 needs about four times its current 92, and a slice that is 23% of the set needs a set four times larger again to hold them — which for a 23% slice at ±0.05 means closer to 1,500 questions overall.

Either size the set for the slices you intend to make decisions on, or stratify deliberately and over-sample the slices that matter, accepting that the overall number then has to be reweighted back to the traffic mix rather than read off directly. Reporting a slice without its interval is how a two-point regression on the majority population gets waved through as noise and a two-point gain on a minority slice gets called a result.

One refinement narrows every band above. The arms run on the same questions, so comparing two of them is a paired comparison rather than two independent estimates, and the quantity carrying an interval is the per-question difference. Questions both arms get right, and questions both get wrong, contribute nothing to it, which makes that interval narrower than the interval on either score. A one-point difference between arms can therefore be readable where a one-point difference from a fixed target would not be. Bootstrap the difference rather than the two levels, and say which you did.

The comparison is also corpus-specific in a way that does not transfer. These numbers describe financial filings, which are figure-dense and template-heavy. A corpus of contracts would show a figure slice near zero and a hybrid arm that costs 7× for nothing — a multiplier readable straight off the cost column, since the hybrid and text-only per-query figures stand in about that ratio, and one that buys nothing once the figure slice empties. Nobody else's three-arm result is evidence about your corpus, and your result stops being evidence about your corpus once the document mix shifts.

The control arm earns its place by answering the question a build decision actually needs, which is one about attribution. Hybrid minus text-only is a number anybody can produce and nobody can act on, since the same six points are consistent with buying evidence and with buying a third vote, and those two purchases have different follow-on costs and different failure modes. Running the arm you would never ship costs one more pass over a question set you already have, and converts an argument about interpretation into a measurement. The slice table does the same job one level down: an aggregate over a set with a majority slice moves slowly by construction, and the capability being bought or lost lives in the rows that aggregate is averaging away.

All of this is aggregate. It tells you the visual arm earns its complexity on this corpus, at this traffic mix, and it says nothing about the specific wrong answer sitting in a support ticket with a customer's name on it. When a user reports that the system misread a figure in a quarterly report, the arm-level table cannot say which stage lost the evidence — and the post-mortem where the parsing owner blames retrieval, the retrieval owner blames fusion and the applied-AI engineer blames the model runs for a week precisely because no one has an artefact that settles it.