Batch deadlines and fallback policy

A timeout that stops waiting converts a slow run into a fast failure, which is an improvement only if somebody downstream can do something with a failure. When the downstream consumer is a dashboard that needs 300,000 enriched rows in a warehouse table, the only thing that produces those rows is work, so a cutoff has to trigger work. The eleven-line version of this — a deadline constant, a comparison against submitted_at, an exception and a pager route — is a reasonable instinct about wanting a deadline attached to the wrong idea of what a deadline is for.

Stopping is also less of an action than it sounds. A batch job runs on the provider's infrastructure, and your poller is a client that reads its status; the job has no knowledge that the poller exists. Raising an exception therefore removes the observer and leaves the work exactly where it was. The provider goes on scheduling the requests it has accepted, goes on completing them, and goes on billing for every one it completes, while the tokens sitting in that job go on occupying the enqueued-token allowance that would otherwise let you submit something else. Cancelling the job is a separate request made to the provider, it is the first line item in the budget below, and a timeout that omits it has stopped nothing except the reading of a status field.

This chapter derives the cutoff properly, working the night backwards from the fixed point. The running workload is a daily product-enrichment job of 300,000 rows submitted to a batch endpoint — the asynchronous interface where a whole collection of requests is submitted at once and collected hours later as a results file — at 22:00, feeding a warehouse table that has to be loaded by 06:00.

Solve for the cutoff, not for the deadline#

Write the night backwards from the thing that cannot move. The dashboard renders at 07:00 from a warehouse table loaded at 06:00. Everything before 06:00 is budget, and the budget has four line items, three of which the previous chapter measured while working through cancellation: the provider's cancel drain at ten to fifteen minutes, the results drain and per-item reconciliation at two to three minutes, the synchronous re-drive at whatever the outstanding set costs in wall clock, and the warehouse load at ten minutes. Only the third varies.

The other three are fixed in the sense that decides the design: they do not move with how much of the batch finished, so they can be added once and carried as a constant. Fifteen minutes for the cancel drain at its slow end, three minutes for the results drain and reconciliation, and ten minutes for the warehouse load come to twenty-eight minutes. The FIXED_OVERHEAD_S of 1,800 seconds in the code below is that sum rounded up to half an hour. Rounding up rather than down is the correct direction, and the reason is that the two errors are not comparable — over-reserving starts the fallback slightly earlier than it needed to start, and under-reserving misses the deadline.

At full volume the worker pool clears 300,000 rows in about an hour, and that hour is a division rather than an estimate. The pool sustains 85 requests per second, and 300,000 rows divided by 85 a second is a little over three and a half thousand seconds, which is fifty-nine minutes. So the naive cutoff is 06:00 minus that hour minus half an hour of everything else, which lands at 04:30 — and 04:30 is the right answer to give if you have to give one number.

It is also more conservative than it needs to be, because it assumes the batch contributed nothing, and on a night when the batch is late it has usually contributed a great deal. That assumption does real damage rather than merely erring on the safe side. At 05:00 on a night where the provider has already served most of the rows, a cutoff that fired half an hour earlier has abandoned a nearly finished job in order to reserve time for work that no longer needs doing, and paid full price for the part it re-drove.

The refinement is that you do not have to guess. Both providers expose per-request counts on the job object while it is still running — Anthropic's request_counts carries processing, succeeded, errored, canceled and expired, and OpenAI's carries total, completed and failed. A poller that already retrieves the job every few minutes therefore knows, at any moment, roughly how many rows the fallback would have to cover.

That such counts exist at all is a property of what a batch job is, and it is the step that makes a moving cutoff possible rather than merely desirable. A batch is a set of independent requests, each of which reaches a terminal state on its own — succeeded, errored, cancelled, expired — and the job as a whole is finished when the last of them has. Progress is therefore a count of settled requests, and a count of settled requests is a quantity that exists long before the last one settles. A single synchronous call has no equivalent, because it is one unit of work with no interior and its only observable states are pending and done. The cutoff can move with the run because the thing being waited on is divisible, which is the same property that made per-item reconciliation possible in chapter 3, where one row is kept per submitted request. Feeding those counts into the deadline arithmetic takes about ten lines:

FALLBACK_RATE = 85          # measured synchronous requests/second, sustained
FIXED_OVERHEAD_S = 1_800    # cancel drain + reconcile + warehouse load, measured

def should_abandon(job, deadline: datetime, now: datetime) -> bool:
    done = job.request_counts.succeeded
    outstanding = job.request_counts.processing + job.request_counts.errored
    fallback_s = FIXED_OVERHEAD_S + outstanding / FALLBACK_RATE
    latest_safe_start = deadline - timedelta(seconds=fallback_s)
    log.info(
        "batch %s: %d done, %d outstanding, fallback needs %.0f min, cutoff %s",
        job.id, done, outstanding, fallback_s / 60, latest_safe_start.time(),
    )
    return now >= latest_safe_start

That function turns a fixed 04:30 into a moving cutoff that tracks how much work is genuinely left. A job that has served 260,000 of its 300,000 rows by 05:00 needs only eight minutes of synchronous re-drive plus the half-hour of overhead, so the cutoff for that job is 05:22 and there is no reason to abandon it at 04:30. A job that has served 40,000 by 04:00 needs fifty-one minutes plus overhead and should be abandoned at 04:39. The same policy, applied to the state of the actual run, buys back an hour of patience on the nights that only need a nudge.

Both of those cutoffs are one subtraction, one division and one more subtraction, and working them is what shows the policy is arithmetic rather than judgement. A job that has served 260,000 of its 300,000 rows leaves forty thousand outstanding, and forty thousand divided by 85 a second is a little over four hundred and seventy seconds, which is the eight minutes. Half an hour of overhead plus eight minutes is thirty-eight minutes, and 06:00 less thirty-eight minutes is 05:22. The bad night runs the same way with a worse count. A job that has served 40,000 leaves 260,000 outstanding, which at the same rate is a little over three thousand seconds, or fifty-one minutes; fifty-one plus thirty is eighty-one minutes, and 06:00 less eighty-one minutes is 04:39.

The calculation makes two choices that are easy to read past. The errored count is added to the processing count because an errored request is a row the fallback still has to produce — it has settled on the provider's side and not on yours — while succeeded rows leave the fallback's work by simply not appearing in it. And the rate is a sustained figure rather than a peak one, because the fallback has to hold it for the whole re-drive; a pool that bursts to twice that number for half a minute and then meets its rate limit clears no more rows in fifty-one minutes than one that never bursts. Laid out against the clock, the bad night looks like this:

Two properties of that timeline are worth stating plainly. The fallback starts before the deadline by exactly the amount of time it needs, which is the definition an exception-and-page timeout is missing. And the batch bar does not stop at the cutoff — it stops twelve minutes later, because cancelling is a request that settles rather than a switch that flips, and a cutoff computed without those twelve minutes fails by exactly twelve minutes on the night it matters.

The belief the whole arrangement is built against is that a batch job has a deadline. It is tempting for a reason that is nearly good rather than for a careless one: the provider publishes a completion window measured in hours, that window is contractual, and a quantity expressed in time and written into documentation reads like a service level. What the window bounds is the point at which the provider stops trying, since requests still unstarted at the end of it are handed back unprocessed. It does not bound the point at which your rows arrive. A poller written against a deadline constant is therefore encoding a promise nobody made, and no value of the constant repairs that, because the quantity the constant is standing in for does not exist.

What firing costs#

The abandonment is not free, and pretending otherwise makes the policy look better than it is. On this workload each row is $0.00725 through the synchronous API and half that on batch, so a clean batch night costs $1,087 and an all-synchronous night costs $2,175. A fallback night sits between them, at a position determined by how much the batch had finished when you gave up.

Every cell in that table is one of two multiplications against those two rates, and the second rate is a little over three-tenths of a cent. The 90 percent row is 270,000 rows at the batch rate, which is the $979, plus 30,000 rows at the synchronous rate, which is the $218, and the night total is their sum. The bottom row is the same pair with nothing on the batch side, so it reduces to the $2,175 already quoted. The last column is each night total less the $1,087 a clean batch night costs, which is why it reads as an overage rather than as a price.

The worst case is bounded, and the bound is the all-synchronous price. That is a genuinely reassuring property: the fallback can never cost more than not having used batch at all, so the money side of "should we fire" has an easy answer, which is that firing is always cheaper than missing the report.

The bound is not a lucky property of these particular figures, and seeing why matters more than the figures do. Every row in the run is billed exactly once, at one of exactly two rates, and the batch rate is the lower of the two. A night's bill is therefore a sum over rows of a per-row price that is never above the synchronous price, so the sum can never exceed what the same rows would have cost with every one of them charged at the synchronous price. Nothing in that argument depends on where the cutoff fired, which is why it holds at every level of completion including none at all, and why the last line of the table is a ceiling rather than a bad case.

The costs that are not on that table are the ones that decide the design. The synchronous pool has to exist, be sized for 300,000 rows an hour, and be in a working state at 04:39 on a Tuesday — which means it cannot be decommissioned, cannot rot, and has to be exercised often enough that its rate limiter, its retry logic and its credentials are known-good. A fallback that fires twice a year is a fallback that will be broken the third time. And the burst itself is real load: 85 requests per second arriving without warning against an account whose limits are shared with other services can trip throttling elsewhere, so the pool needs the headroom reserved rather than merely available.

There is a data-shape cost as well, and it is the one that surprises people the first time a fallback fires successfully. The run now has two sources — some rows enriched by the batch job at 02:00 and the rest by the synchronous pool at 05:00 — and unless the loader writes both through the same reconciliation-driven path, the warehouse ends up with a partial load from one source and a full overwrite from the other. Keeping the fallback's outputs flowing into batch_items, the per-item state table from chapter 3, under the same run_id with the same states and the same coverage metric, is what stops a fallback night from being a second class of run that nothing downstream knows how to reason about. The fallback is a different transport for the same work, not a different pipeline.

The slip rate that ends the argument#

"At what point is the discount not worth it" has a tempting answer and a correct one. The tempting answer is a break-even calculation, so run it: across a 30-night month, at various slip rates, assuming a slip night is abandoned at 40% completion.

Those monthly figures are thirty nights of one of two kinds, added up. A clean night is $1,087, so a month of them is $32,625, and thirty all-synchronous nights at $2,175 each would be sixty-five thousand two hundred and fifty dollars — which is where the last column comes from, since it is that figure less the month's spend. A slip night abandoned at 40 percent completion costs $1,740, which is $653 more than a clean night, so two slip nights add about thirteen hundred dollars to the month and six add about four thousand. Each row is the clean-night total with that many nights swapped for the more expensive kind.

The dollars never flip. Even a job that is abandoned every single night is cheaper than one that never used batch, because the rows completed before each cutoff were bought at half price. If cost were the only axis, the answer would be to keep the batch path forever regardless of how badly it performs, and that answer is wrong.

What actually degrades is the shape of the operation. At one slip in fifteen the batch path is primary and the fallback is an exception: it fires occasionally, someone notices, and the cost is a modest overage. At one slip in two the fallback is the primary path in everything but name — it produces half the rows and it carries the deadline — and yet it is treated as the emergency branch, tested by accident, and triggered by a policy that fires at 04:39. Somewhere between those two the design has stopped being a batch pipeline with a safety net and become a synchronous pipeline with an unreliable pre-stage, and the honest response is to relabel it accordingly.

Relabelling it changes the design rather than the endpoints. Instead of an exception path guarded by a cutoff, submit the batch at 22:00, run the reconcile-and-re-drive unconditionally at 04:30 every night, and let it re-drive whatever is outstanding — nothing on a good night, everything on a terrible one. The batch endpoint becomes a best-effort cost optimiser whose contribution varies with the provider's queue, the deadline is met by a path that runs every day and is therefore always working, and there is no 04:39 decision for anyone to get wrong. The monthly bill lands in the same range as the table above. That trade — a slightly lower savings ceiling in exchange for a single always-exercised path — is usually the right one once slips are regular.

Constants that go stale#

FALLBACK_RATE = 85 and FIXED_OVERHEAD_S = 1800 are the two numbers the entire policy rests on, and both are properties of a system that changes: a pool tuned six months ago, a warehouse load that got slower as the table grew, a cancel drain nobody has timed since the provider changed its scheduler. A cutoff computed from stale constants is a wrong one in an unknown direction, so re-derive both from the last few fallback runs on a schedule.

The other standing temptation is to route the cutoff to a human instead of to code. A page at 04:39 asks somebody to invent a plan with fifty minutes of budget remaining, and the plan they invent will be the one the policy should have encoded, executed more slowly. Notify by all means. Do not make the notification the mechanism.

A deadline on a batch job is a claim about when work must be finished, and a cutoff is a claim about when the alternative has to start; conflating the two produces eleven lines of code that fire on time and accomplish nothing. Derived properly, the cutoff is the deadline minus a fallback duration that the provider's own counters let you recompute every five minutes, and firing it is bounded above by the price of never having used batch at all, which is what makes the arrangement safe to run unattended.

All of which is a lot of machinery — durable ids, latency measurement, per-item reconciliation, cancellation, cutoff arithmetic, a fallback path kept warm — for a workload that has to earn it. The final question is which workloads do, and the sorting rule most teams reach for turns out to be the wrong one.