412,000 dead letters beside a green dashboard describe an ownership gap

Two numbers can be true about the same ingest path at the same time and point in opposite directions. One is a rate: the share of a day's documents that failed to parse. The other is an age: how long the oldest unrepaired failure has been sitting in a queue. A rate is bounded, it is cheap to alert on, and it is what dashboards are built to display. An age is unbounded and almost nobody puts one on a dashboard. When the two disagree, the age is the one carrying information, and the reason is arithmetic rather than judgement.

Quarantine, or a dead-letter queue, is the store an ingest path writes a document into when it has failed in a way that retrying will not fix, so the document is set aside rather than dropped. Coverage, throughout this chapter, means the share of a source's documents that are actually present in a search index. A quarantine record is what makes a coverage figure computable at all: without one, an absent document leaves no trace, and the only way to count what is missing is to enumerate the source and diff it against the index.

Why a flat rate cannot move#

Put the dashboard's number into symbols, because the whole argument is visible there and is invisible anywhere else. The published rate is the count of documents quarantined in a day divided by the count of documents processed in that day. Look at what that expression does not contain: any term at all for how many documents are already sitting in the queue. Multiply numerator and denominator by the same factor and the ratio does not move. Hold the ratio pinned at any positive value for a year and the queue grows by that value times the denominator, every day, without limit.

That is the general shape rather than a complaint about one dashboard. A defence was installed — a threshold on a rate — and the property it was installed to catch is a total. The two are independent by construction, so the rate can be held perfectly constant while the total goes anywhere it likes. An alert that has never fired in a year is therefore not evidence of health; it is evidence that no threshold was ever crossed, and the only threshold configured was on a ratio that cannot move when the numerator and denominator grow together.

Age behaves the opposite way, which is what makes it useful. The oldest open record gets one day older every day unless somebody closes it, and no change in traffic, in mix or in denominator can slow that down. A quantity that increments on its own until a human acts is the only kind of signal that reports inaction. A queue that has been accumulating for eleven months has therefore never been drained, which means the drain procedure has either never been written or never been executed, and an unexecuted recovery procedure should be assumed not to work.

Read the numbers before touching anything#

The first useful action is a set of queries against the quarantine table, and they take about ten minutes. The three below ask different questions in a deliberate order: how much of the queue is distinct work rather than repetition, what the distinct work is made of, and what any of it costs the people using the index.

-- How much of this is distinct documents rather than repeated attempts?
SELECT count(*) AS records,
       count(DISTINCT (source, doc_id)) AS documents,
       min(first_seen_at) AS oldest,
       max(first_seen_at) AS newest
FROM quarantine;

-- Which cohorts is it made of?
SELECT parser_version, error_class, error_fingerprint,
       count(DISTINCT doc_id) AS docs,
       min(first_seen_at)::date AS since
FROM quarantine
GROUP BY 1, 2, 3
ORDER BY docs DESC
LIMIT 20;

-- What does this cost us in coverage, per source?
SELECT q.source,
       count(DISTINCT q.doc_id) AS missing,
       d.indexed,
       round(100.0 * d.indexed / (d.indexed + count(DISTINCT q.doc_id)), 2) AS coverage_pct
FROM quarantine q JOIN source_totals d USING (source)
GROUP BY q.source, d.indexed;

On the pipeline in question those returned 412,000 records covering 61,400 distinct documents, which is the first correction to the story — the queue is not 412,000 problems, it is roughly 61,000 documents that have been failing on every sync since the day they first failed, plus a long tail that failed once. The cohort query returned a shape that is typical: 43% of the distinct documents last failed under pdfminer.six@20221105, a parser version retired four months ago; 22% carried error_class = permanent_input with an encrypted-PDF fingerprint; 9% were resource_fatal on files over 100 MB; and the remaining quarter spread across two dozen fingerprints with fewer than 400 documents each. The coverage query said the wiki source was 99.4% indexed and the contracts source, where the PDFs live, was 92.8%.

The gap between the first two figures is a ratio worth computing rather than reading past. 412,000 records over 61,400 documents is a little under seven attempts per document on average, which is what a queue looks like when a connector re-offers the same failing documents on every sync and nothing ever closes them. The headline count is inflated by roughly that mean attempt count, the work to be done scales with the documents rather than with the records, and any alarm set on the record count is measuring sync frequency as much as it is measuring breakage.

The cohort shares close on a whole. Forty-three, twenty-two and nine leave a remaining quarter with nothing left over, and what licenses treating the four as a partition rather than as overlapping sets is the word last: every document has exactly one parser version it last failed under, so no document is counted twice. That is also what will license scaling a canary result from one cohort to the rest of that cohort, and no further.

The tail does not close as neatly, and the gap is informative. A quarter of 61,400 is around sixteen thousand documents, while two dozen fingerprints holding fewer than 400 each account for at most about ten thousand. The cohort query returns only its top twenty rows, so the tail runs past what was listed: either there are more fingerprints than two dozen, or some of them hold more than four hundred documents. The query as written cannot say which, and a real investigation would re-run it without the limit before believing either.

The contracts coverage figure is the one the dashboard should have been showing. A parse failure rate is a property of a day's traffic; coverage, meaning the share of a source's documents that are actually present in the index, is a property of the corpus, and it is the only number that corresponds to what a user experiences when the thing they are looking for is not there. The two diverge exactly the way a bug close rate diverges from an open bug count — a team can close bugs at a perfectly respectable rate for a year while the backlog doubles, and no rate metric will tell them.

That comparison is an analogy, and it breaks in the place that matters. A bug backlog has a front page: the open count is one saved filter away, inside a tool the team already opens for other reasons. A quarantine backlog has no such page. The coverage figure that plays the part of the open count does not exist until somebody writes the join above and puts it somewhere, which is why the analogy understates how easily this state persists.

The 0.4% figure was also measured against the wrong denominator. It counted quarantines against all documents processed, including the 3.1M markdown and HTML pages that essentially never fail, which dilutes a 7% failure rate on contracts down to something that looks like noise. Failure rates want to be reported per source and per content type, because that is the granularity at which the cause lives and the granularity at which a threshold is meaningful.

The mechanism of that dilution is worth being explicit about, because it is not an error in the measurement. A rate over everything is a weighted average of the per-source rates, and the weights are each source's share of a day's traffic. A source with a high rate and a small share contributes almost nothing to the average, so a large body of pages that essentially never fail can hold the published figure near zero no matter what any individual source does. The number is not wrong. It is answering a question about the average document, and nobody was asking one.

One numeral does double duty here, and the two jobs are not the same job. The 7% named just above as a failure rate on contracts is a flow: the share of a day's contract documents that fail. The other side of that source's 92.8% coverage is a stock: the share of it absent from the index right now, accumulated over eleven months and never once reduced. A flow and a stock can coincide numerically, and these two are close enough to be read as a single figure. Nothing here establishes what relationship holds between them, so carry them separately, and have the contracts failure rate measured on its own before either is used in an argument.

The drain that is safe, and the one that is not#

Whether to drain at all is a real question, because there is a cheaper-sounding alternative: rewind the contracts connector's cursor to zero and resync the whole source, which re-fetches all 880,000 documents and repairs whatever the current parser can now handle without anybody writing drain code. That costs about 49 hours at 5 documents per second against the source API, several million chunks of embedding work, and it produces no cohort information — documents that still fail simply fail again and land back in the queue, so you finish knowing exactly what you knew, having spent two days of API budget. Draining by cohort touches 61,400 documents instead of 880,000, runs in a few hours, and yields the breakdown that tells you which parser fixes are worth making. The resync is the right tool when the failure was in the connector's coverage rather than in individual documents; here the queue already names the documents, so the targeted operation dominates.

Both durations come out of one division. Eight hundred and eighty thousand documents at five a second is a hundred and seventy-six thousand seconds, which is the forty-nine hours above. Sixty-one thousand four hundred at the same rate is a little over twelve thousand seconds, which is the few hours. The ratio between the two is the ratio between the two document counts — the targeted drain touches roughly one document in fourteen — and every other cost in the comparison, chunks and tokens and money, scales by that same factor.

The obvious drain is a script that reads every record, feeds the stored payload to the current parser, writes what succeeds and deletes what does not. It is wrong in three separate ways, and the third one is not recoverable.

Replaying stored payloads replays the past. A record captured in April holds bytes fetched in April, and the source document may have been edited nine times since, or deleted. Writing April's content into the index in March of the following year is a stale write, which chapter 3's version guard would reject if the record carries a source_version and would happily accept if it does not. Beyond correctness, it wastes the work: the current revision is the one anybody wants, and fetching it costs the same as reading the payload from S3.

Replaying against the current parser without holding the versions fixed makes the results uninterpretable. A record that succeeds might mean the parser has been fixed, in which case the other 26,000 documents in its cohort are also repairable and should be scheduled immediately, or it might mean somebody re-exported that one contract from a tool that produces cleaner PDFs, in which case nothing has been learned. Comparing the record's source_version against the source's current etag at replay time distinguishes them, and reporting the two outcomes separately is what turns a drain into information about the parser.

Deleting the residue destroys the only evidence that those documents were ever meant to be in the index. After the delete, the pipeline's state is indistinguishable from a pipeline that never attempted them, the coverage query above returns 100%, and the next person to ask why a contract is missing starts the investigation from nothing. The residue is the answer to a question the organisation will ask again.

The coverage claim in that paragraph has a mechanism behind it in the SQL, and it is worse than losing evidence. The coverage query computes its figure by joining source_totals against whatever is in quarantine, so the missing count it subtracts comes entirely from the queue. Empty the queue and there is nothing to subtract. It is an inner join, so a source with no quarantine rows left drops out of the result set altogether, and a dashboard reading that result shows either full coverage or no row at all — both of which a person reads as no gap here. Deleting the residue does not only remove the record of the loss. It turns the one metric that would have detected the loss into a metric that reports perfection.

The drain that holds up runs by cohort, re-fetches from the source, and lands behind the guards the earlier chapters put in place:

def drain_cohort(fingerprint, parser_version, *, limit=500, rate=5):
    rows = quarantine.select(fingerprint=fingerprint, parser_version=parser_version, limit=limit)
    outcome = Counter()

    for row in rows:
        current = source.head(row.doc_id)
        if current.status in (404, 410):
            quarantine.close(row, disposition="deleted_at_source")
            outcome["gone"] += 1
            continue

        changed = current.etag != row.source_version
        try:
            process_document(source.fetch(row.doc_id))   # version guard applies inside
            quarantine.close(row, disposition="repaired_content_changed" if changed
                                              else "repaired_parser_fixed")
            outcome["repaired_changed" if changed else "repaired_parser"] += 1
        except IngestError as e:
            quarantine.record_attempt(row, exc=e)
            outcome[f"still_failing:{e.fingerprint}"] += 1

        time.sleep(1 / rate)

    return outcome

The throttle in that loop is a defence, and it is worth asking what it actually defends. rate is spoken of as requests per second, and time.sleep(1 / rate) puts a fifth of a second between iterations, so what it caps is documents per second. Requests per second is a different quantity, and it varies underneath the cap in two ways. Each surviving document costs two requests against the source, a head and then a fetch — and a ceiling written in requests per second counts both by name — so the request rate is twice the document rate. A document that is gone at the source costs one request and then hits continue, which jumps past the sleep entirely, so a cohort of documents deleted at the source runs with no throttle on it at all.

The cap also binds only when the work is slow. The achieved document rate is whatever the per-document work allows, up to five a second and no higher, and the canary below ran far under that. The closer the work gets to free — a warm cache, a parser that fails fast, a cohort of small files — the closer the drain climbs to its cap, and five documents a second is ten requests a second, which is the whole of the source's ceiling with nothing left for the live connector. That is what makes the knob a trap rather than an inaccuracy: it is set once, on the belief that it bounds pressure on a shared source, and the pressure it produces rises exactly when the drain starts running well.

Running that over a 500-document canary of the largest cohort — a deliberately small first run, taken from the same population, whose result stands in for what the full run will do — took nineteen minutes and returned 71% repaired_parser, 6% repaired_changed, 4% gone and 19% still failing under two new fingerprints. That result justifies scheduling the full 26,000-document cohort and predicts its outcome closely enough to plan the load: 26,000 documents at 5 per second is 87 minutes of source API traffic, about 104,000 chunks, roughly 42M embedding tokens, and something under a dollar of inference — all of it competing with the live connector for the same rate limit, which is why the drain runs at a fixed low rate rather than as fast as it can.

Three of those figures check each other. The four dispositions sum to a whole with nothing unaccounted for, which is what licenses scaling them at all: every document in the canary landed in exactly one bucket, so the proportions carry across to the rest of the cohort. The cohort size is the earlier breakdown run on the same document count — forty-three percent of 61,400 is the 26,000 above — so the two queries agree without sharing a step. And the chunk figure divides back out to four chunks per document, which is the number a re-chunk of this cohort would have to reproduce for the token estimate to hold.

The nineteen minutes and the eighty-seven minutes are not the same kind of number, and dividing one into the other gives a wrong answer. Nineteen minutes across five hundred documents is wall-clock time, and it contains the fetch, the parse, the chunking and the embedding — a little over two seconds a document, of which the throttle accounts for a fifth of a second. Eighty-seven minutes is a budget: 26,000 documents at the configured five a second is how much of the source's request allowance the drain consumes, not how long it will be running. Scaled from the canary's wall-clock, the same cohort occupies many hours. Both figures are correct about the resource each measures, and only the second is denominated in the unit a rate limit is written in.

Cohorts whose canary repairs almost nothing get closed rather than retried. An encrypted-PDF cohort is not going to change until somebody decides whether the pipeline should hold decryption credentials, which is a product decision with a security review attached, so those 13,500 documents move to a register of known-unindexable content with a reason code and a named owner, and they stay counted in the coverage number. A corpus that is 92.8% indexed with a documented reason for the other 7.2% is in a defensible state; the same corpus reported as 100% indexed is not.

That register's size is the second cohort share applied to the same document count: twenty-two percent of 61,400 is the 13,500 above. Moving them changes nothing about their absence from the index, because a register is a disposition and not a repair, which is exactly why they have to stay inside the coverage figure. A number that improves when documents are reclassified is a number that can be improved by reclassifying documents.

The drain also has to yield to the live path rather than compete with it. A source API with a 10 requests-per-second ceiling is already carrying the incremental connector at 2 to 4 requests per second during working hours, and a drain that consumes the remainder will start producing 429s that the connector's retry logic absorbs by falling behind. Running the drain at a fixed 5 requests per second overnight, or having it read the connector's current consumption from a shared token bucket and take only the slack, keeps the freshness guarantee intact while the repair runs. The same applies to the embedding provider's quota, where a drain that saturates the account's tokens-per-minute limit delays every fresh document by however long the drain takes.

Both of those ceilings are counted in requests and in tokens. The knob in the loop above is set in documents, and a document is two requests and four chunks, so the two units have to be converted before the fixed rate can be claimed to fit inside the slack. The shared token bucket is the version that does not need the conversion, because it reads the quantity the ceiling is written in.

Thresholds that can actually be crossed#

The alerting that prevents this state is on quantities that grow when the problem grows.

Two of those rows are one quantity at two cut points. Coverage below 99% means more than one percent of a source is missing; more than 0.5% of a source's total sitting in quarantine is the same measurement at half the cut. The second row therefore fires strictly before the fifth can, and the fifth can never fire on its own. On this pipeline's own figures the two disagree about one source: the wiki at 99.4% indexed is missing six tenths of a percent, which trips the distinct-documents row and leaves the coverage row quiet, while the contracts source at 92.8% trips both. The fifth row is not a second detector. It is the first one restated in the unit somebody outside the ingest team can act on.

Read the rest of the table against the state it was written for. The age row would have fired within a week of the first record and gone on firing for eleven months. The new-fingerprint row would have fired once as each fingerprint appeared. The drain-job row would have fired eight days in. The row that would have stayed quiet the entire time is the fourth, because a quarantine rate compared against its own 30-day median cannot detect a leak that has been steady for a year — the median moves with the leak. The row shaped most like the dashboard that failed fails in the same way, and it earns its place for a different job: catching a sudden change, which is the one thing the others are slow at.

The last row exists because a recovery procedure that is not exercised does not work. Running the drain weekly on whatever is in the queue keeps the code honest, keeps the credentials valid, and keeps the runtime of a full drain within something a person will actually authorise.

Retention forces disposition. A record older than 90 days is either repaired, closed as deleted at the source, or moved to the unindexable register with a reason — and the transition is not automatic deletion, since deletion is the failure mode this whole chapter is arguing against. Making the expiry a task that appears in a queue somebody owns is the mechanism; the number on the dashboard is only a symptom of whether the mechanism exists.

Ownership has to be as concrete as the thresholds. The queue needs a team name in whatever service catalogue the organisation keeps, an entry in that team's on-call runbook describing the drain, and a standing item in a weekly review where the coverage figure per source is read out loud. The failure mode this pipeline demonstrated is not that anybody decided to ignore 412,000 records; it is that the records were not on anybody's list, and a metric nobody reads is functionally identical to a metric nobody emits. Publishing coverage to the teams that consume retrieval rather than only to the team that runs ingestion is what makes the number stick, because the consuming team notices a drop from 99.4% to 96% in a way the owning team, watching a flat rate, did not.

Where coverage stops being computable#

There is a limit to what any of this buys. A scanned fax with no OCR budget behind it — optical character recognition, the step that turns an image of a page into text a parser can read — is going to stay unindexed, and the honest end state for a real corpus is a coverage figure in the high nineties with a documented tail. Reporting that figure in the product — a note that this source is 92.8% indexed, with the gaps enumerable — is more useful to the people relying on retrieval than any dashboard that reads green.

A second limit sits inside the coverage query itself. It joins source_totals, which means somebody has to be able to state how many documents a source holds. A source that offers a change feed and no count gives you a numerator with no denominator, and coverage for it is not a number that can be computed at all — only the quarantine count is available, and a raw quarantine count is the record-versus-document trap from earlier in this chapter. Where a total cannot be obtained, the age of the oldest open record is the signal that still works, because it needs nothing from the source.

A queue hundreds of thousands of records deep and a green dashboard above it are consistent with each other, and reading them as a contradiction is the first mistake. The rate was doing exactly what a rate does. What was missing was a quantity that grows when the problem grows, a threshold set on that quantity, a name written against it, and a procedure that had been executed at least once. Every fix in this chapter is one of those four, and not one of them is a change to a parser.

Everything in this chapter rewrites a bounded set of documents, a few tens of thousands at a controlled rate, while the index keeps serving. The rewrite that does not fit that description is the one that touches every document in the corpus at once.