Bounded repair loops

Repair is what happens after a response has failed and before the document is given up on: another call, possibly with different parameters, possibly with the validator's complaint fed back, up to some cap. It is the last gate in an extraction service and the easiest one to reason about badly, because every individual attempt is cheap and the counter it moves goes in the direction everybody wants.

The two questions worth answering before choosing a cap are where the attempts actually land in a system's latency budget, and what the documents that exhaust the cap are failing for. Both have answers that a per-attempt cost calculation cannot reach, and both point at the same conclusion: the cap is rarely the variable that matters.

Where extra attempts actually land#

The documents that exhaust a cap are not a random sample of traffic. They are the largest documents in the stream, because output length is what pushes a response into truncation, and a response that truncates takes longer than one that does not — it generates all the way to the ceiling before stopping. On a document-extraction service running against a p99 budget of 4 seconds, meaning 99 out of every 100 documents have to finish inside that, overall p50 latency is 1.2 seconds for a response of roughly 450 output tokens, and a response that runs to a 1,500-token ceiling takes about 3.1 seconds. Every attempt spent on an exhausting document is therefore a 3.1-second attempt rather than a 1.2-second one.

The 9.3 seconds at the foot of that diagram is the cap multiplied by that slower figure — three attempts at 3.1 seconds each — and the multiplication is where the cap stops being a count. Three attempts on an ordinary document would have held a slot for less than four seconds in total. Three on an exhausting one hold it for more than two and a half times that, and every one of those three is known in advance to be a full-ceiling generation, because that is what put the document in this cohort. So the number a cap really allocates is slot-seconds, and it allocates them at the worst exchange rate the system offers.

Those documents already sit past the 99th percentile, so the direct effect on the reported p99 of doubling their attempt budget is close to nil; a request at the 99.6th percentile is invisible to a statistic computed at the 99th. The cost lands somewhere else. Every one of those attempts holds a worker slot and a slice of the provider's throughput allocation, and the pool is shared with the 99.6% of documents whose latency the budget is actually about.

The arithmetic that matters is Little's law — the queueing result that the concurrency a system needs equals its arrival rate multiplied by the time each item occupies a slot — applied over the window when these documents are concentrated. Truncation clusters at month-end, when the large multi-page invoices arrive: over that window the repair-eligible fraction rises from 6% to about 19% and the exhausting fraction from 0.4% to 5%, against an arrival rate at the peak of roughly 70 documents per second.

Running the law on the table's own numbers is two multiplications, and it turns the last row from a claim into a consequence. At a cap of three the mean end-to-end service time over the window is 2.31 seconds, and 70 documents a second each occupying a slot for 2.31 seconds needs 162 slots. At a cap of six the mean service time rises to 2.77 seconds, and the same arrival rate needs 194. The pool holds 160. The first configuration is a hair under capacity and the second is over it by more than a fifth, and the distance between those two outcomes was opened up by a change touching 0.4% of documents.

Once required concurrency exceeds the pool, documents queue before their first call, and queue wait is added to every document that arrives during the window, including the 81% that would have been served in 1.2 seconds. That is the mechanism by which a change affecting 0.4% of traffic shows up in the p99 of the rest, and it is invisible in any calculation that treats the extra attempts as independent of everything else running.

The non-selectivity of that wait is what does the damage, and it is worth one more step. While required concurrency sits below the pool size, a document is served the moment it arrives and its latency is its own service time. Once the requirement passes the pool size there is no free slot at the moment of arrival, so a document waits before its first call — and the wait is not selective, because a queue has no idea which document is large. Every arrival during the window inherits it, the 81% that would otherwise have been served in 1.2 seconds included. The slow documents are not the ones that suffer; they were already past anybody's budget.

What the exhausted documents are failing for#

The stronger objection needs the four-way failure split applied to the documents that exhausted the cap rather than to traffic as a whole.

Eighty-three percent of that pool is failing for a reason that is a property of the request rather than of the sample, so more attempts at the same parameters are more full-price ways to be told the same thing. The two classes that do respond to another attempt respond because something genuinely differs between attempts: sampling is stochastic, so formatting damage may not recur, and a schema failure can be retried with the validator's message appended, which changes the input.

That eighty-three percent is the truncated 61% plus the refused 22%, and the two belong together for a reason more precise than both being bad. A truncated document failed because the request carried a ceiling and the document needed more than the ceiling; send the identical request again and the ceiling is still there and the document is still that size. A refusal failed because of what the input is; send the identical request again and the input is still what it was. In both, the cause is a property of the request, and drawing a different sample has no purchase on it. The remaining seventeen percent is exactly where a different sample does have purchase, which is why those two classes are the ones a repair loop is for.

That distinction is what a repair loop should be written against. A loop that counts attempts treats all four classes alike; a loop that reads the class stops immediately on the two where stopping is correct, and changes a parameter on the one where a parameter is the cause.

MAX_ATTEMPTS = 3
TOKEN_CEILING = 4000

def plan_next(outcome: Outcome, attempt: int, prev: Attempt,
              errors: list[ValidationError]) -> Attempt | None:
    """Return the next attempt, or None to stop. Class-aware, not count-aware."""
    if attempt >= MAX_ATTEMPTS:
        return None
    if outcome is Outcome.REFUSAL:
        return None                       # answered by declining
    if outcome is Outcome.TRUNCATED:
        if prev.max_tokens >= TOKEN_CEILING:
            return None                   # identical call, identical ceiling
        return Attempt(max_tokens=min(prev.max_tokens * 2, TOKEN_CEILING),
                       feedback=None)
    if outcome is Outcome.SCHEMA:
        return Attempt(prev.max_tokens, feedback=render_errors(errors))
    if outcome is Outcome.PARSE:
        return Attempt(prev.max_tokens, feedback=PARSE_NUDGE)
    return None

The loop needs one bound above the per-document cap, which backend engineers already run for flaky HTTP dependencies under the name of a retry budget: repair attempts may not exceed 10% of primary call volume in any five-minute window, and when they do, repairs stop and failures pass straight through to the dead-letter queue. Without it, a bad prompt deploy at 09:00 turns the repair loop into a 4× traffic amplifier aimed at a provider that is already the reason everything is failing, and the pipeline recovers minutes later than it would have if it had simply given up.

The four comes straight from the cap: one primary call plus up to three repairs is four calls per document, so a deploy that fails every document multiplies request volume by four at the exact moment the provider or the prompt is already the bottleneck. The budget is written as a share of primary volume rather than as an absolute rate because a share stays meaningful when traffic changes, while a fixed ceiling of so many repairs a minute is either useless at peak or a brake at trough. And the action on exhaustion has to be to stop repairing rather than to slow down, because a repair loop that queues under load is adding latency to documents that were going to fail anyway while holding slots the healthy ones need.

Three numbers, and how each one is usually measured wrong#

A repair loop is worth keeping only if it is instrumented on three axes, and each of the three has a definition that flatters the loop and a definition that tells the truth.

Recovery rate is commonly reported as the fraction of repair attempts that succeed, which rises whenever you add attempts to easy cases and is unrelated to how many documents make it through. The number to hold is the fraction of documents that failed their first call and were eventually recovered, reported per attempt index so the marginal contribution of each attempt is visible. On this service the marginal recoveries run 70%, 17%, 7%, and then 8% spread over attempts four through six, and a curve that flat past attempt three is the argument against the sixth attempt on its own.

Reading those four figures needs one clarification: they are shares of the documents that were eventually recovered, split by which attempt did the recovering, so between them they account for essentially all of the recoveries and what matters is the shape rather than any single value. The first attempt takes most of what is there. The second takes most of the remainder. By the third the curve has flattened, and three further attempts together contribute less than the second one did on its own. That shape is what a stochastic cause looks like as it exhausts itself — documents whose failure was a bad sample get a fresh sample early, and what survives two or three tries is the population whose cause never depended on the sample. Which is the same population the class breakdown identifies, arrived at from the other direction.

Added latency should be measured on the whole population, not on the repaired one. The repaired documents' own latency is uninteresting because it is already outside anybody's SLO; what the budget cares about is the shift in the pool's p95 and p99 during the windows when repairs cluster, which means the metric has to be sliced by hour rather than averaged over a week.

Token cost per recovered document is the number that usually decides the argument, and it is the one most often replaced by cost per attempt. Cost per attempt is small by construction and stays small however badly the loop performs; dividing total repair spend by documents actually recovered is what surfaces a loop paying more per document than the humans it is trying to spare.

The repair that makes the parser happy and the record wrong#

A different proposal usually arrives alongside a cap increase, and it is cheaper and faster than any of this: a normalisation step in front of the parser that strips trailing commas, quotes bare keys, and closes unbalanced brackets. Somebody measures it and recovery goes from 94% to 99%.

Two of those three rules are safe and one is not, and the difference is whether the repair recovers information that is present in the bytes or invents information that is absent. A trailing comma is a syntax artefact sitting next to a complete value; removing it changes nothing about what the document says. An unquoted key is the same. Closing an unbalanced bracket writes the one token the model never produced, and the reason it never produced it is that generation stopped.

{"invoice_number": "INV-90412", "issue_date": "2024-11-03",
 "total": 1240.00, "currency": "USD", "status": "partial",
 "line_items": [{"description": "Rack unit 2U", "quantity": 2, "unit_price": 89.50},
                {"description": "Patch cable 3m", "quantity": 12, "unit_price": 4.20},
                {"description": "Shelf brack

Append "} ] } to that and the result parses, validates against a strict Pydantic model, and writes. The row carries a total of 1240.00 and three line items summing to 229.40, on an invoice that had forty. The parse counter does not fire because it parsed, the schema counter does not fire because every required field is present with the right type, the truncation counter does not fire because the repair ran before anything looked at finish_reason, and the document arrives in the ledger indistinguishable from a clean extraction. The 5 percentage points of extra recovery are made of records like this one, which is why the measurement looked good.

The 229.40 is worth adding up by hand, because the size of the error is the argument. Two rack units at 89.50 is a hundred and seventy-nine; twelve patch cables at 4.20 is a little over fifty; the two together are the 229.40. The invoice had forty line items, so thirty-seven of them are gone, and the total of 1240.00 is the only surviving trace that anything else was ever on the page. Nothing downstream recovers them, because nothing downstream knows they are missing. Whether the repaired record clears a strict validator depends on how much of the half-written item the repair keeps — closing the truncated string leaves an item carrying a description and no price, which a contract requiring a price on every line item would reject, while discarding the partial item altogether leaves a well-formed array of complete items with nothing to object to. The tidier repair is the more dangerous one, which is the usual direction for this kind of fix.

The rule that keeps the safe repairs and drops the dangerous one is to gate normalisation on the discriminators rather than on the parser's opinion: run it only when finish_reason is not length, never synthesise a closing token, and tag every record the normaliser touched so the rate is countable and the affected rows are findable later. A repair you cannot count is a repair you cannot roll back.

Exhaustion needs a landing place that survives a replay#

Something has to happen to the documents that fail at the cap, and the shape of that path is what makes the whole loop safe to re-run. Writing a partial record is the option to rule out first, since a row with nulls in it is a row that downstream aggregation will treat as data; the useful fallback either writes nothing or writes a row in a needs_review state that every consumer query excludes by construction.

Idempotency, the property that applying an operation twice leaves the system in the same state as applying it once, matters because the dead-letter queue exists to be drained twice. Somebody fixes the prompt, replays a day of failures, and half of them succeed — and the replay must not produce a second invoice for a document that was already written, or a second review task for one already in a reviewer's queue. Keying both the write and the dead-letter entry on a hash of the document bytes together with the prompt and schema versions gives an upsert — a write that inserts the row or overwrites the existing one rather than adding a second — that is stable across replays, and makes "reprocess everything that failed under prompt v7" a query rather than an archaeology project.

None of this is a substitute for the earlier gates. A repair loop is the last thing to reach for, after constrained decoding where the endpoint supports it, after a prompt that stopped asking for a justification paragraph, after a max_tokens ceiling that matches the document distribution. It is also not the mechanism for provider outages, which are the non-2xx class and belong to backoff and circuit breaking, on a different code path with a different owner. What the loop is for is the residue: the stochastic formatting damage and the near-miss schema failures that a second sample genuinely fixes, bounded so that the failures it cannot fix arrive quickly, counted, and reproducible.

That is the whole apparatus for this topic — three request modes with three guarantees, a scan that finds one complete value or rejects, a versioned validator, four discriminated failure classes, four counters, and a bounded loop with an idempotent floor. It produces exactly three outcomes for any document: a validated object, a counted failure with a class attached, or a dead-letter entry with a stable key. What it does not produce is a correct object, and the nine subtotal errors in the audited hundred and twenty from the opening chapter are still sitting in the ledger, well-typed and wrong, waiting for the business rules that know what an invoice is supposed to add up to.