A paper is worth reading for the one claim you could falsify

A staff engineer on an analytics team spent a Friday afternoon with a paper on candidate re-ranking and wrote it up in four hundred words for the team wiki. The summary was accurate. It said the authors sample multiple candidate programs from the model, score each with a verifier, and return the highest-scoring one, and that this improved execution accuracy — the share of generated queries that run and return the rows the question asked for — on a public text-to-SQL benchmark by 12.4 points. Three weeks later the team had shipped a re-ranking stage into their question-to-SQL service, and in the retro nobody could say whether it had worked. The dashboard showed accuracy at 73.1% against 71.3% before, which was inside the noise band of their eval set, and the argument in the room was about whether the verifier was undertrained or the sampling temperature was wrong or the benchmark had never resembled their traffic. All three were plausible. The summary had left no way to tell them apart, because it recorded what the paper said rather than what the team should expect to see.

That wiki page is one instance of a job that arrives constantly and is almost never taught. A research paper is a written report of an experiment: what the authors built, how they measured it, what the measurement came to, and what they claim it means. In machine learning most papers reach you as preprints, versions posted by the authors themselves to arXiv — an open repository anyone can upload to, usually before peer review and frequently instead of it — so nothing upstream has filtered the claims before they arrive at your desk. Several hundred that touch a working engineer's stack appear in a month, and they reach an engineering team through a colleague's link in a channel, a conference talk, a vendor's release notes citing three of them, or a director who read a summary of a summary and wants to know why the team is not doing it.

Reading research, as a working practice rather than an academic one, is the job of turning that stream into decisions about your own system: which single idea inside a paper could change how your service behaves, whether the evidence behind it is strong enough to act on, and what finding out would cost. It exists because both of the obvious alternatives are expensive. Adopting whatever circulates most widely converts novelty into roadmap churn and leaves production carrying machinery that never earned its place, while ignoring the literature entirely means paying for the year it takes a genuinely useful technique to reach you by osmosis. Most of the practice, by volume, is deciding what to leave alone with a reason you could give out loud. The summary in the story above is a competent piece of comprehension that cannot support a decision, and the distance between those two things is what the rest of this chapter is about.

The gap is not effort. The engineer read carefully and understood the method. What the write-up never contained was a statement that could turn out to be false. "Re-ranking improves accuracy" cannot be false in any useful sense, since some workload somewhere will always show a gain; it is a description of a mechanism dressed as a prediction. A claim you can act on has to be specific enough that a particular measurement, taken on your system, would embarrass it.

Four things a usable extraction names#

The extraction worth writing down has four parts, and it is the fourth that most notes are missing.

The mechanism is what actually happens at runtime, stated in the vocabulary of your own system rather than the paper's. Not "self-verification improves reasoning" but "for each question, sample eight candidate SQL statements at temperature 0.8, run each through the database's EXPLAIN planner to discard the ones that do not parse or reference missing columns, score the survivors with a small classifier trained on query-result pairs, and return the top one." Written that way, the mechanism already tells you which parts of your service change and roughly what they cost.

The concrete change is the diff. One new service call between generation and execution, an extra 7 model calls per question, a classifier that has to be trained on something and served somewhere, and a fallback for when all eight candidates fail the planner check. Naming the diff is what converts a paper into an estimate of engineering weeks.

The metric that should move is a number you already collect, on a dataset you already have, with a magnitude attached. "Execution accuracy on the 480-question internal eval set, currently 71.3%, should reach at least 76% if the mechanism transfers at half the reported strength." Half the reported strength is a deliberate discount and the reasons for it are the subject of the next few chapters; the point here is that a target with a number in it can fail, and "should improve" cannot.

The failure condition is the sentence that says under what circumstances the mechanism ought to do nothing. Re-ranking can only help on questions where the candidates disagree and where the wrong candidates are distinguishable from the right one. If the model already produces the same query in all eight samples, re-ranking has nothing to choose between; if the failures are silently wrong queries that parse cleanly and return plausible numbers, the planner check cannot see them and the classifier is being asked to do a much harder job than the paper's verifier was. Both of those are properties of your traffic, and both are measurable before any of the mechanism is built.

The failure condition usually names a cheap pre-test#

Stating when something should fail tends to hand you an experiment that costs an afternoon rather than a sprint. In the re-ranking case, the failure condition is about candidate disagreement, so the pre-test is to sample eight candidates for the existing eval set and measure how often they disagree at all.

from collections import Counter

def disagreement_profile(questions, sample_fn, k=8):
    """Sample k candidates per question; report where re-ranking could possibly help."""
    stats = Counter()
    for q in questions:
        candidates = [normalise_sql(sample_fn(q, temperature=0.8)) for _ in range(k)]
        distinct = set(candidates)
        if len(distinct) == 1:
            stats["unanimous"] += 1                    # re-ranking is a no-op
        elif any(c == q.gold for c in distinct):
            stats["gold_present_but_not_top1"] += 1    # headroom for re-ranking
        else:
            stats["gold_absent_from_all"] += 1         # re-ranking cannot fix this
    return stats

Run that over 480 questions at roughly 3,900 model calls and a few dollars of inference, and the answer arrives before anyone opens an editor. Suppose it comes back as 61% unanimous, 14% with the correct query present but not selected first, and 25% where no sample is correct. The ceiling on re-ranking is then 14 points of execution accuracy in the impossible case where the verifier is perfect, and a verifier that is right 70% of the time on those contested cases while never damaging a correct top-1 answer buys about 9.8 points. That is a real upside, and it is also a number the team can compare against the cost of eight samples per question. If instead the profile had come back 88% unanimous, the ceiling would have been under 5 points and the project would have been over before it started.

The same move works on claims that have nothing to do with sampling. A paper on retrieval reranking fails where your top-1 retrieved chunk is already the right one, so measure recall@1 against recall@10 first. A paper on long-context attention fails where 94% of your requests are under 4,000 tokens, so plot the token-length histogram first. A paper on speculative decoding fails where your workload is dominated by prefill rather than generation, so measure the input-to-output token ratio first. In each case the pre-test is a query against telemetry you already have, and it either opens the project or closes it for a day's work.

Two papers, extracted the same way#

Extraction pays off most when several candidates are competing for the same engineering week, because the four fields turn them into rows.

The second row is where most of the argument usually happens, and the fourth row is where it should. Notice that the schema-linking candidate has a failure condition that can be checked with a catalogue query: if the median database in production has 11 tables and the whole schema already fits in the prompt with room to spare, the mechanism is addressing a constraint the team does not have. Papers are written against the constraints of the authors' setting, and the failure condition is where those constraints become visible.

Reading in passes, and stopping on purpose#

Extraction of this kind does not require reading the paper front to back, and the practice that survives contact with a real backlog is a multi-pass one — the approach NVIDIA's "How to Read Research Papers: A Pragmatic Approach for ML Practitioners" lays out for engineers, and which the "Ten simple rules for reading a scientific paper" guidance frames as reading with a question in hand rather than reading to absorb.

The first pass is title, abstract, figures and the results table, and it answers one question: is there a mechanism here that could change something in my system. Most papers stop here, and stopping is the correct outcome rather than a failure of diligence. The second pass reads the method section closely enough to write the mechanism sentence in your own vocabulary, and skips related work, most of the introduction, and every experiment that is not on a task resembling yours. The third pass, which perhaps one paper in twenty earns, reads the appendix — where the prompts, the hyperparameters and the excluded cases live — and it is where the failure condition usually becomes clear, because appendices are where authors record what they had to control.

A useful discipline is to write the mechanism sentence before the third pass and then check whether the appendix contradicts it. If the method section describes a verifier and the appendix reveals the verifier was fine-tuned on 40,000 human-labelled examples from the same benchmark, the mechanism sentence has to change, and with it the change-to-the-system row and the estimate in weeks.

What this displaces is the habit most engineers bring from university, which is to read linearly with a highlighter and produce a faithful précis at the end. That habit was appropriate when the goal was to be examined on the content, and it scales badly against a field publishing several hundred relevant preprints a month. Reading with a question already in hand means most papers are dismissed in ninety seconds by their own results table, which is the only way the volume is survivable at all. The engineers who describe filtering the arXiv firehose — as in "How I Read AI Research Papers as a Software Engineer" — are all describing some version of the same triage, where the scarce resource is not reading time but the willingness to stop.

There is a second-order benefit to writing the extraction down that has nothing to do with the paper. Six weeks later, when someone in the channel asks whether the team ever looked at candidate re-ranking, the answer is a file with a date, a mechanism sentence, a pre-test result and a target, rather than three people's differing recollections of an afternoon. Papers recirculate constantly, often under new names and with new authors, and a repository of extractions is what stops the same evaluation being performed four times by four people who each believed they were first.

What this is not#

Extraction is not a summary, and the difference matters enough to be worth stating plainly. A summary is faithful to the paper and is judged by whether it represents the authors correctly. An extraction is faithful to your system and is judged by whether the prediction it contains turns out to be right. Both are legitimate documents; only one of them can be wrong in a way that teaches you something. If your team keeps a paper channel, the summary belongs in the channel and the extraction belongs in the repository next to the eval set, with a date on it.

Extraction is also not reproduction. Nothing in this chapter runs the paper's code or checks whether the authors' number is real, and the disagreement profile above tests your workload rather than their claim. That ordering is deliberate, since a mechanism that cannot help your traffic does not need its evidence audited.

There are papers from which no falsifiable claim can be extracted, and that is not an indictment of them. Surveys, position papers, dataset releases and analyses of model behaviour are all doing something other than proposing a mechanism you could adopt, and forcing a prediction out of them produces a bad prediction. The honest note for those is a line saying what the paper is for and why it does not imply a change, which stops the same paper being re-litigated in the channel every few months.

The limitation of the whole exercise is that the failure condition is a hypothesis about your own system, not a proof. The re-ranking pre-test measures disagreement under one sampling temperature on one eval set that somebody assembled a year ago, and if that eval set over-represents the tidy questions people asked during the pilot, the profile is measuring the pilot rather than production. An extraction can be wrong in exactly the way it invites you to catch, which is the property that makes it worth writing.

Having a target of 76% assumes the paper's 12.4 points were themselves worth discounting from rather than worth ignoring. That number came from a comparison the authors chose, against a baseline they configured, and the most common way such a comparison misleads has nothing to do with honesty.