Top-1 on a screenshot test rewards resemblance, not evidence

A screenshot evaluation set is the ordinary way to decide whether a visual retriever is good enough to ship. It is a collection of page images, a question written against each of them, and a rule for turning rankings into a number. Top-1 is that rule in its commonest form: the fraction of questions for which the single highest-ranked result is the page the question was written from. That page is the positive, or gold, unit — the one result the scoring treats as correct. Every other page the retriever could have returned is a negative, also called a distractor.

What that definition leaves out is the answer to the question. It asks whether one particular page came back first, and a page can come back first for reasons that have nothing to do with holding the fact somebody asked for. Two choices made while assembling such a set decide almost everything the resulting number will mean: which pages get to compete against the answer, and what the word "correct" is taken to cover. Both fall back on something convenient when nobody chooses deliberately, and both conveniences are generous to the system under test.

A negative drawn at random can be rejected on appearance#

Consider the commonest way one of these sets gets built. An engineer takes 500 page renders sampled across a corpus of 400,000 pages, shows each to a vision model, and asks it to write a question answerable from that page. The page it came from is the gold unit. Everything else in that 400,000-page collection is a negative. Scoring top-1 then asks: given this question, does the retriever rank the source page above all 399,999 others?

That sounds like a demanding test, and the arithmetic of it — one page against four hundred thousand — is what makes it sound demanding. The difficulty of a ranking task is not set by how many negatives there are, though. It is set by how much the hardest negative resembles the positive, because a negative that is easy to reject contributes nothing to the ordering no matter how many copies of it there are. Four steps carry that from a property of the corpus to a property of the score. A page drawn uniformly at random from four hundred thousand pages of filings and decks shares almost nothing with a segment-revenue bar chart. So a single coarse feature — is this a chart about revenue at all — separates the positive from nearly every negative in the pool. So a ranking decided by that one feature already puts the gold unit first, and the score reports only whether the encoder learned it. And a feature that coarse is learned by any encoder worth deploying, which is why the number lands near the top of its range for every candidate system and stops telling them apart.

The question generated from a segment-revenue bar chart is something like "which segment contributed the most revenue?", and the competition is the entire corpus — audit opinions, risk-factor prose, signature pages, org charts, cash-flow tables. The retriever has to distinguish "a revenue chart" from "not a revenue chart". That is a real capability and it is a much easier one than the product needs, because in production the competition is not the whole corpus. It is the other 400 quarterly decks that contain a segment-revenue bar chart with the same template, the same colours, and different numbers.

The distractor set is the experiment#

A retrieval metric is only as discriminating as the negatives it scores against. With random negatives, a system that has learned "this query is about a revenue chart" wins. To find out whether it has learned anything more, the negatives have to be pages where that heuristic is useless.

The construction rule is: for every question, the distractors should be units that share everything with the answer except the fact being asked for. In practice that means grouping the corpus by visual template, then selecting siblings that differ only along the axis the question probes.

Read that table down its two right-hand columns rather than row by row, and read it as a gap rather than as a set of levels. The gap between the two columns is the whole experiment. On rows B through E the columns disagree — the page looks exactly like the answer and does not contain it — and a system ranking on appearance cannot separate those rows from row A. On row G the columns agree, so any system at all separates it from row A.

Only candidate G is the kind of negative the original test set was scoring against. Candidates B through E are the ones production actually contains, and every one of them is a plausible top-1 return for a system matching on appearance. Rows B and D differ from A only in period; row C only in metric; row E only in reporting entity. Those three axes — period, metric, entity — cover the large majority of near-miss retrievals on business corpora, and they are enumerable from document metadata rather than requiring judgement.

Generating them can be mostly mechanical:

def hard_negatives(unit, corpus, k=6):
    """Units that share this unit's visual template but differ on exactly one axis."""
    siblings = corpus.by_template(unit.template_id)     # same layout signature
    pools = {
        "period": [u for u in siblings
                   if u.entity == unit.entity and u.metric == unit.metric
                   and u.period != unit.period],
        "metric": [u for u in siblings
                   if u.entity == unit.entity and u.period == unit.period
                   and u.metric != unit.metric],
        "entity": [u for u in siblings
                   if u.metric == unit.metric and u.period == unit.period
                   and u.entity != unit.entity],
    }
    out = []
    for axis, pool in pools.items():
        for cand in nearest_by_appearance(pool, unit, n=k // 3):
            out.append({"unit_id": cand.unit_id, "axis": axis, "label": "distractor"})
    return out

Each pool holds two of the three axes fixed at the gold unit's own values and lets the third vary, which is why the three filter expressions each carry two equality tests and one inequality. Inside a pool the candidates are then ordered by appearance and the nearest few taken, so the distractors that survive are the ones that both differ on the intended axis and look most like the answer.

Where a corpus has no such metadata, the same pools can be built by clustering page embeddings and taking within-cluster nearest neighbours, which is cruder and still far better than random. The point is not the specific grouping key; it is that a distractor which the retriever can reject on appearance alone tells you nothing.

That last sentence is a requirement and the three axes are not, and the two are worth keeping apart. The requirement is that the negatives be pages a retriever cannot reject on appearance, and it follows directly from what the metric is supposed to discriminate: a measurement can only report on a distinction its inputs contain. Period, metric and entity are required by nothing. They are a design decision, taken because business documents carry all three in their metadata and so the pools can be built without human judgement. A corpus of laboratory reports or maintenance photographs would need different axes, and the requirement above would be unchanged.

An analogy helps here, and its limit matters as much as the picture does. A set with random negatives is a multiple-choice paper whose wrong options were lifted from a different exam: anybody with a passing acquaintance with the subject scores well, and the paper separates nobody. The picture breaks at one point worth naming. A multiple-choice paper has a small fixed number of options, chosen by an examiner who can make the paper as hard or as easy as they like. A retriever ranks whatever the organisation happens to hold, so the difficulty of the real task is set by the corpus and not by anyone's choice. Hardening the negatives is therefore not making the exam unfairly hard. It is making the exam resemble the corpus it was supposed to be about.

Questions that no distractor can answer#

Hard distractors immediately expose a second defect, this time in the questions. Run the set above against the question "which segment contributed the most revenue?" and candidates B, D and E all answer it — they each contain a segment revenue chart with a largest bar. The question is under-specified, so there is no unique correct retrieval, and any metric computed against a single gold unit is scoring an arbitrary choice.

The reason this happens is structural rather than careless, and it is worth following through. A vision model writing a question is shown one page and nothing else. Nothing in its input describes the rest of the collection. So no property of the question it produces can depend on which other pages exist. So a question that happens to be unique to its page is unique by luck, and a question that four hundred pages answer is under-specified without anybody being able to tell from the page.

Vision models writing questions from a single page have no idea the corpus contains four hundred similar pages, so they produce period-free, entity-free questions constantly. In one 500-question set built the naive way, 61% were answerable from at least one distractor.

The validation step is direct: for each candidate question, check whether any hard distractor can answer it. If one can, the question needs a qualifier or it leaves the set.

def is_well_posed(question, gold_unit, distractors, judge):
    if judge.answers(question, gold_unit) is not True:
        return False, "gold does not answer its own question"
    for d in distractors:
        if judge.answers(question, d) is True:
            return False, f"answerable from distractor on axis={d['axis']}"
    return True, "ok"

The first test in that function is the one people leave out, and it is the cheaper of the two. A question the gold unit cannot answer is a broken item however good the distractors are, and it happens whenever a vision model writes about something it inferred rather than something the page states. Only after that passes does the second loop mean anything, because a question that its own gold unit cannot answer would be rejected by every distractor too, for the wrong reason.

Questions that survive look different from the ones that went in. "Which segment contributed the most revenue?" becomes "Which segment contributed the most revenue in FY24 Q3 for the consolidated group?" That is more verbose than a real user query and it is the version that measures anything, because it is the version with exactly one correct answer in the corpus. If you want the evaluation to also cover under-specified user queries, that is a separate slice with a separate metric — a set of queries where several units are acceptable, scored on whether any acceptable unit was retrieved — and it should not be silently mixed into a set claiming top-1 accuracy.

Score sufficiency, not similarity#

The second half of the fix is what gets counted. Top-1 asks whether the gold unit came back first. What the product needs is whether the retrieved set can support a correct answer, and those diverge as soon as more than one unit could contribute.

Label every candidate in the pool with a three-way judgement rather than a binary one:

• sufficient — a competent reader could derive the answer from this unit alone
• supporting — relevant and mentions the subject, insufficient on its own (chapter 1's OCR'd chart page is the canonical example: topically right, evidentially empty)
• irrelevant

Then report sufficient@k: the fraction of questions for which at least one retrieved unit in the top k is labelled sufficient. Reporting the supporting@k rate alongside it is worth the extra column, because a system with high supporting and low sufficient is precisely the failure this topic opened with, and the two numbers together name it without further investigation.

On the same index, the same 500 questions, and the same encoder, the two evaluations report this:

Before subtracting anything across that table, look at its bottom row, because it says the two columns do not share a denominator. The right-hand column is computed after 305 items have left the set, so the left-hand column describes five hundred questions and the right-hand one describes what survived validation. Subtracting a figure in one column from the figure beside it is therefore not available: the difference would mix a change in the negatives, a change in the labels and a change in the question set into one number that names none of them. The two columns answer different questions about the same system, and that is the honest reading of them.

The bottom row also closes exactly against a figure from earlier in the chapter, and the closure is worth stating because it tells you what the row is not. Sixty-one hundredths of five hundred is three hundred and five, with no remainder. The 305 in the table is the 61% figure multiplied by the size of the set — one measurement written in two units, rather than two measurements agreeing. Nothing in the table corroborates the 61%; a reader who takes the row as independent confirmation has counted the same observation twice.

What can be subtracted is within a column, because a column keeps its denominator fixed all the way down. In the left-hand column, going from one slot to five buys about six points. In the right-hand column the same four extra slots buy about twenty-four. That difference is a diagnosis rather than a curiosity. Under random negatives the first slot was already right almost every time, so there was nothing left for slots two to five to recover. Under hard negatives, roughly a quarter of all questions have their sufficient unit sitting somewhere in ranks two to five — present in the pool, and not first. A system whose recall is fine and whose ordering is not is a ranking problem, and it points at the step that decides ordering rather than at the encoder that decides membership.

Two rows in the right-hand column also close against each other. A question is counted in supporting-only@5 exactly when its top five holds something relevant and holds nothing sufficient, so the two rows are mutually exclusive by construction and cannot both count the same question. Adding them gives the fraction of questions whose top five contains anything relevant at all, and that sum falls a little short of one: roughly one question in twenty-five comes back with five results that are neither sufficient nor supporting. That residual is small and it is real, and it bounds what a pure recall improvement can win. Almost every question already retrieves something on-subject. The deficit is evidential, which means it has to be paid for in representation or in ordering rather than in reach.

Nothing about the system changed. The 0.91 was never wrong as a measurement of what it measured; it was answering "can this retriever tell a revenue chart from an org chart", and the answer is yes. The 0.43 answers "can this retriever find the specific evidence for this specific question in the presence of look-alikes", and that is the question the product poses on every request. The supporting-only@5 figure of 0.29 says that on nearly a third of queries the top five contained something topically right and evidentially empty and nothing better — which is a direct, per-query count of the failure chapter 1 described anecdotally.

The mistake to name here is not a slip in anybody's arithmetic. It is the most dangerous shape a measurement can take: a quantity that stays constant while the property you care about varies across its whole range. Top-1 against random negatives returns the same score for a retriever that can only tell a revenue chart from an org chart and for one that can pick the right quarter, the right metric and the right legal entity out of four hundred look-alikes. Discrimination among siblings goes from none to perfect, and the reported number does not move. Every instinct that treats a stable metric as a trustworthy one is inverted here, because stability under a change that should have moved the number is the signature of a measurement that is not connected to it.

The ceiling this measurement sets on answer quality#

It is a retrieval metric, and retrieval metrics bound answer quality from above without predicting it. sufficient@5 of 0.67 means at most 67% of these questions can be answered correctly from evidence; how many actually are depends on resolution, prompt composition, and the model, all of which chapter 3 showed can throw away evidence that retrieval delivered. Reporting sufficient@k as if it were product accuracy is the same category error as reporting recall@5 was, one level up.

The reason it is a ceiling and not an estimate follows from what the label means. A unit marked sufficient is one from which a competent reader could derive the answer, which says the evidence was present in the retrieved set. Whether it reaches the model at a legible resolution, whether it survives prompt assembly, and whether the model reads it correctly are three further conditions, and each of them can only subtract. There is no route by which a question with no sufficient unit in its top five gets answered from evidence, so the metric can never be exceeded and can be missed by any margin.

It is also not a substitute for public benchmark numbers, and public benchmark numbers are not a substitute for it. Document-retrieval benchmarks tell you how one encoder compares to another on their corpora and their query distributions, which is genuinely useful when choosing an encoder and tells you nothing about whether your fastener tables are distinguishable from each other. Encoder selection and system evaluation are different activities that happen to produce similarly-shaped numbers.

The cost of labelling, and how to keep it tractable#

The honest cost is annotation. Tri-labelling ten candidates per question takes an annotator who understands the domain roughly five to seven minutes per question, so a 500-question set is on the order of 50 hours of expert time — and it has to be partly redone whenever the corpus grows enough to introduce new look-alikes. Three things make it tractable. Pool the candidates: label the union of what several system variants retrieved, once, so the labels are reusable across experiments rather than per-run. Use a model to pre-label and have humans adjudicate; on this material model-human agreement on the sufficient/supporting boundary ran around 0.84, which is high enough to save most of the work and too low to skip the audit, so report the agreement rate alongside the results rather than burying the fact that a model helped. And keep the set small and adversarial rather than large and representative — 200 well-posed questions with hard distractors discriminate between system variants far better than 5,000 questions with random negatives, because the easy questions are answered identically by every variant and contribute nothing but confidence intervals.

That last recommendation is the one people argue with, because a larger set is the standard remedy for a number nobody trusts. It is the wrong remedy here, and the reason is worth stating plainly rather than being left to intuition. Adding questions narrows the interval around an estimate; it does not move the estimate. The defect in a random-negative set is a defect in what its number measures rather than in how precisely it measures it, and every question added inherits that defect intact. Five thousand questions of the easy kind buy a very precise measurement of whether the encoder can tell a chart from a page of prose.

Two routes in this chapter arrive at nearly the same set size by arguments that share no step. Discarding every question a sibling can also answer leaves, out of five hundred, a handful short of two hundred; asking how many items it takes to separate two system variants recommends about two hundred. One is a fact about how under-specified vision-written questions are on this corpus, the other a fact about how much two candidate systems differ, and the agreement between them is a coincidence of this material rather than a law — a corpus of more distinctive pages would discard far fewer questions while needing the same number to discriminate. What it does license is a practical note: here, validation is not throwing away a set you needed, it is arriving at the size you were going to choose anyway.

Growing the set from production rather than from sampling#

A set built by sampling pages measures the corpus. A set built from failures measures the product, and the two diverge quickly once real traffic arrives. Every support ticket about a wrong answer is a question with a known-correct evidence unit and a known-incorrect retrieval, which is exactly the shape of an evaluation item — the expensive part, deciding which unit is sufficient, has already been done by whoever investigated the ticket.

The discipline is to convert each one before the ticket closes: record the question in the user's phrasing rather than a cleaned-up version, record the unit that should have been retrieved, generate the hard distractors for it, and add it to a regression slice scored separately from the main set. Reported separately, that slice answers a question the sampled set cannot — whether the failures users actually hit are getting fixed — and it accumulates the specific look-alikes your corpus produces, which are always stranger than the period/metric/entity axes anticipate. On this filings corpus the regression slice surfaced a category nobody had modelled: restated prior-period figures, where the same entity, metric and period appear twice in the corpus with different values and only a footnote distinguishing them.

That last category is the clearest case of why the axes are a design decision rather than a discovery. A restatement holds period, metric and entity all fixed and varies only the revision, so every one of the three generators above returns an empty pool for it and the mechanical route produces no distractor at all. Production found the axis that metadata could not. A regression slice is the part of the evaluation that can learn a new axis, and the sampled set is the part that cannot.

A retrieval score is a statement about the negatives it was computed against, and nothing more. Ninety-one percent against pages that could be dismissed on sight and forty-three percent against siblings that could not are both correct measurements of the same index, and they differ because only one of them was asked a question the product asks. The three moves that convert the first into the second are cheap next to what they reveal: build the distractors so that appearance cannot separate them from the answer, throw out the questions more than one of those distractors satisfies, and score whether the answer can be derived from what came back rather than whether the intended page came back first.

Run this evaluation against a hybrid system and a specific, reproducible pattern falls out of the per-query traces. On the questions whose evidence is in a chart, the image candidate is labelled sufficient, is present in the candidate pool the image retriever returned, and is absent from the final top ten. Retrieval found it. Something between retrieval and the prompt discarded it, and that something is the step where three ranked lists with three incompatible scoring scales get merged into one.