Your training pairs already exist in logs somebody is currently deleting

Training a retrieval model requires examples of the relation it is supposed to learn, and that relation is narrower than it sounds. A positive pair is one query together with one passage that genuinely answers it, which is the unit every method in this topic consumes. It is not a relevance judgment. It says this passage is relevant to this query, and it says nothing about whether it is the best passage, whether other passages are equally relevant, or whether the passages nobody chose are irrelevant.

That narrowness is what makes the data cheap, because a system already in production generates the relation as a byproduct of working. Somebody types a question in their own words, something is returned, and they act on one of the results. The pairing is produced by the person who had the problem, in the vocabulary they actually use, without an annotator being briefed on the domain. It is noisier than a judged label and it is not free of problems, but the problems are known, nameable and mostly correctable, and the alternative is usually a stalled project.

Where a positive pair hides#

The generative relation to look for is any record of somebody asking and something answering. Search logs are the obvious instance: a click is a person deciding that a passage was worth opening after reading a query they wrote themselves. Support tickets are a second: the customer's description is a query written in customer language, and the knowledge-base article or resolution note that closed the ticket is a passage that answered it. Chat and assistant transcripts are a third, especially where the system already cites its sources, because a cited chunk in an answer the user did not follow up on is a weak positive and a cited chunk in an answer the user thanked is a strong one. Internal documentation carries a fourth, structural source: FAQ headings paired with the body beneath them, runbook titles paired with their procedures, changelog entries paired with the issue that prompted them.

The last two columns are one fact written twice. Each bias is a statement about which part of the relation the source got wrong, and each correction removes exactly the offending part rather than adjusting a weight — discard the ranks where position dominates, take the customer's message instead of the agent's, keep only turns carrying a signal the retriever did not produce. That is why the table has no row whose correction is "tune it down". A bias that comes from which records exist is fixed by choosing different records, and the one row where that does not work is the last, whose correction takes a chapter.

The yields matter as much as the biases, because the four sources are complementary along an axis nobody notices until the fine-tune underperforms: coverage. Click logs cover the head of the query distribution and almost none of the corpus, since 200,000 documents cannot all have been clicked in a month. Doc structure covers the whole corpus uniformly and none of the query distribution. A training set built only from clicks teaches the model to be better at the 3,000 documents people already find, which is precisely the region where retrieval was working.

Neither number in the yield column tells you that. A source producing 15–40k pairs a month and a source producing one pair per heading are being compared on a quantity that says nothing about which passages the pairs point at, and the failure it causes appears only after training, as a fine-tune that improved a benchmark and not the system.

The extraction is a query, not a project#

Pulling pairs out of a click log is a SQL problem with three filters on it, and writing those filters is where the quality of the training set is mostly decided.

WITH clicked AS (
  SELECT
    s.query_text,
    c.chunk_id,
    c.result_rank,
    c.dwell_seconds,
    s.session_id,
    ROW_NUMBER() OVER (PARTITION BY s.session_id ORDER BY c.clicked_at) AS click_seq
  FROM search_events s
  JOIN click_events c ON c.search_id = s.id
  WHERE s.created_at >= now() - interval '30 days'
    AND length(s.query_text) BETWEEN 8 AND 300
),
kept AS (
  SELECT query_text, chunk_id, result_rank
  FROM clicked
  WHERE dwell_seconds >= 20          -- drop pogo-sticking: opened, bounced
    AND click_seq = 1                -- first click only; later ones are browsing
    AND result_rank BETWEEN 2 AND 50 -- deliberately skip rank 1
)
SELECT query_text, chunk_id, count(*) AS support
FROM kept
GROUP BY query_text, chunk_id
HAVING count(*) >= 2                 -- at least two independent users agreed
ORDER BY support DESC;

The dwell_seconds >= 20 filter is the cheapest quality gain available, and the reason runs through what a short dwell means. A click followed by a return to the results page in three seconds is evidence the passage looked relevant in the snippet and was not, which makes it closer to a hard negative than a positive. Training on it as a positive therefore teaches the model to reproduce exactly the mistake the user corrected, which is worse than dropping the row and worse than never having had it. On the corpus described here, the dwell filter removed 22% of clicks and shifted mean judged precision of the surviving pairs from 0.71 to 0.86 when 300 of them were hand-checked.

The result_rank BETWEEN 2 AND 50 line is the aggressive form of position-bias correction and deserves the argument behind it. Users click the first result at a rate far above its actual relevance, because it is first; the click therefore carries much less information than the same click at rank 12, where the user scrolled past eleven alternatives to reach it. The principled treatment is inverse propensity weighting, where each pair's contribution to the loss is scaled by 1/p(click | rank), with the propensities estimated from a small randomised swap experiment. The blunt treatment is to discard rank-1 clicks entirely, which throws away roughly 45% of your data and requires no experiment. Start blunt, because a fine-tune whose training set is dominated by "the model already ranked this first" is being taught to keep its current opinions, and that failure is invisible in the loss curve.

The HAVING count(*) >= 2 line trades volume for reliability. Requiring two independent sessions to have made the same query-chunk association removed the long tail of one-off clicks and cut 21,000 raw pairs to 8,900, which sounds like a loss and is not: published fine-tunes at this scale routinely reach their gains on 5,000 to 20,000 pairs, and the marginal pair is worth much less than the marginal error is worth avoiding. Cap the head as well, because 400 distinct queries generated 12,000 of those clicks; taking at most 5 pairs per unique query string brought the set to 6,200 pairs across 2,850 distinct queries, which trains a model that can generalise rather than one that has memorised the top of the traffic distribution.

That head figure is worth dividing out rather than passing over. Four hundred query strings account for twelve thousand of the clicks, which is well over half the raw pairs coming from a set of queries small enough to read in an afternoon. Without a cap, more than half the gradient would come from a few hundred phrasings, and a model has no way to know that a phrasing repeated ten thousand times is one query rather than ten thousand pieces of evidence. The cap is not a tidying step; it is what stops the head of the distribution from being the training set.

Assistant transcripts deserve the most suspicion of the four, because the citation in a generated answer was chosen by the retriever you are trying to improve. Training on those pairs teaches the model to keep retrieving what it already retrieves, which is the purest form of the feedback loop that makes a system confidently worse over time. The salvage is to keep only turns carrying an independent signal — the user marked the answer helpful, the ticket closed without escalation, the code snippet was copied — and to discard the rest.

Even then the candidate set was drawn by the current model, and that is the part no filter reaches. A transcript can only contain passages the retriever returned, so a pair mined from one is silently conditioned on the retriever having succeeded. The source therefore looks equally healthy whether your retriever is excellent or badly broken: in both cases every pair it yields is a pair the retriever already gets right, and the population you care about — the passages it currently fails to retrieve — cannot appear in it at all. A source whose output does not vary with the thing you are trying to measure is not a weak source of evidence about it; it is no evidence at all.

If none of these tables exist, the first commit of this project is the one that creates them, and it is small. A search_events row wants the raw query string, the user's role or segment, the timestamp, the ranked list of returned chunk ids, and the model and index version that produced them; a click_events row wants the search id, the chunk id, the rank it occupied, the click time and the time the user came back. The model and index version are the fields people leave out and later need, because a training set spanning a retriever change is two datasets with different biases stacked on top of each other. Retention is the other decision to make now: a 30-day policy is a security default that nobody chose for this purpose, and a hashed, query-only table retained for a year is usually acceptable where the full event log is not.

That last point is worth handling before the data leaves the database rather than after. Query strings from a support product contain account numbers, names, serial numbers and occasionally passwords typed into the wrong box, and a training set is a file that ends up on a GPU host, in a bucket, and inside model weights that are hard to unlearn. Run a scrubber over the query column, drop queries matching a credential pattern outright, and keep the training file under the same access controls as the log it came from. The cost of this is an hour; the cost of skipping it is a model that occasionally emits a customer's account number as a nearest neighbour.

Tickets need a different join and one discipline. The pair is the customer's first message and the article that resolved the ticket, never the agent's closing summary. An agent writes "escalated to firmware; replaced the MX-4471-B controller board per KB-2210", which shares vocabulary with the knowledge base because both were written by the same population of people, so a model trained on it learns to match internal language to internal language and gets no better at the customer who wrote "the unit clicks twice and the green light never comes on". Vocabulary mismatch between the query side and the passage side is the entire problem adaptation is meant to solve, and pairing agent text with agent text quietly removes it from the training data.

The lineage of the technique#

This is how the base model you are fine-tuning was built. MS MARCO, the dataset underneath a large share of publicly available retrieval encoders, is a million real questions sampled from Bing's query logs with human-selected passages attached; the questions read like questions because people typed them into a search box while trying to accomplish something. The BEIR benchmark that followed measured how well models trained on that log-derived data transferred to eighteen other domains, and the general finding — strong in-domain, uneven out of it — is the reason this topic exists at all. Web query logs are not your query logs, and the gap between them is what you are closing.

There is a limit to how far the analogy carries, and it is worth stating because it is the difference between the two datasets rather than a caveat about analogies in general. MS MARCO's passages were judged by humans after the fact, so the dataset has both scale and a cleaning pass; your click log has scale and no cleaning pass. Treat the filters above as that pass, and hand-check a sample of 200 pairs before training anything, because the check takes an afternoon and catches the categorical mistakes — a chunk_id join that drifted after a re-index, a template query fired by a monitoring bot 4,000 times, a period where the search UI was broken and clicks were random.

The most consequential mistake with log-derived pairs is to use them as the evaluation set as well, and it is tempting because they are abundant and free. The golden set from stage 8.3 exists precisely because it is a stratified, hand-judged, deliberately constructed sample; log pairs are none of those things and share their biases with the training data, so a model that overfits the biases scores well on both. Keep them physically separate: a train table and a judged table, no query string appearing in both, and the judged set frozen against the snapshot it was built on.

The weakness of a positive pair as a claim also becomes load-bearing later. Because it says nothing about the passages the user did not click, "not clicked" and "not relevant" remain different statements, and a pipeline that conflates them when negatives are mined in chapter 6 manufactures wrong labels at scale.

None of this helps the team whose search box was shipped last month, whose ticket system is a shared mailbox, and whose corpus is 200,000 documents nobody has ever queried. That team has no logs to mine and still needs pairs, and the standard answer is to generate questions from the documents themselves with a model. It works, it is cheap, and it has one specific weakness that determines whether the resulting fine-tune measures anything real.