Batch pricing and the completion window

Two numbers decide whether a batch endpoint suits a piece of work: what it charges and when it finishes. The first is written on a rate card and the second is not written anywhere, and treating them as though they were the same kind of fact is how a plausible one-paragraph proposal turns into a missed deadline. A batch endpoint — Anthropic's Message Batches API, OpenAI's Batch API — is the asynchronous interface to the same models, where you submit a whole collection of requests at once, get back a job identifier rather than an answer, and collect a results file some hours later.

This chapter is about pricing that trade honestly, and about turning "will this be back by 06:00" from an argument into a measurement. The worked example is a review-scoring job — two million archived product reviews that need a model-assigned quality score — because it is large enough for the discount to matter and ordinary enough that the arithmetic transfers.

What the discount actually covers#

Two things, and it is worth being precise about both because neither is "faster" and neither is "more reliable".

The first is the price cut. Anthropic's Message Batches API and OpenAI's Batch API both bill at 50% of standard input and output rates, and the discount applies to every token in the job — no minimum volume, no negotiation, no separate contract. On the review-scoring workload that is straightforward arithmetic once you have measured the token shape of a sample. Scoring one review took roughly 800 input tokens (the review text plus a fixed instruction block) and produced about 120 output tokens (a score, a label, a one-line justification).

Those per-review figures become job totals by multiplication, and the multiplication is allowed because each review is a separate request billed on its own tokens. A batch pools nothing, and no rate on either card moves with how many requests happen to arrive together. Two million reviews at 800 input tokens each is one thousand six hundred million input tokens, and two million at 120 output tokens each is two hundred and forty million output tokens. Written in millions, which is the form the rate cards want, the job for two million reviews on claude-opus-5 comes to 1,600 million input tokens and 240 million output tokens. Both rate cards quote a price per MTok, one million tokens, which is the unit the table below uses.

Every cost cell in that table is one multiplication and every total is one addition, and both are worth performing rather than reading. On the synchronous side, 1,600 MTok at $5.00 per MTok is $8,000, since a price quoted per million tokens multiplied by a count already expressed in millions leaves currency and nothing else. The output side, 240 MTok at $25.00, is $6,000. Input and output are the only charges in the job, so the total is their sum, $14,000. The batch column is the identical pair of multiplications against halved rates: $2.50 and $12.50 give $4,000 and $3,000, which sum to $7,000.

Halving each rate and halving the finished bill are the same operation here, and the reason they are the same is a property of this particular discount rather than of discounts generally. The price of the job is a sum of terms, each of which is a token count multiplied by a rate, and the discount applies per token with no threshold and no tier. Multiplying every rate by one half therefore multiplies every term by one half, and a sum of halved terms is half the original sum. A discount with a minimum volume, or one that applied only above some level of spend, would not distribute across the terms that way, and for those the two operations would come apart.

The two rates differ by a factor of five, and where that factor comes from decides which half of a job's bill deserves attention. Input is read in one pass over the whole prompt, and the positions in that pass are worked on together rather than one after another. Output cannot be produced that way, because each token the model writes depends on the tokens it has already written. Every output position therefore needs its own pass of the model, and those passes have to run in sequence, so a token the model writes costs several times what a token it reads costs and the rate cards are pricing that difference rather than pricing "tokens" as one commodity. The consequence on this workload is that the output column is nearly as large as the input column while carrying about a seventh of the volume — 240 MTok against 1,600, and $6,000 against $8,000 — which is why a job's token shape, and not only its token count, is what sets its cost.

The second thing you buy is the disappearance of throughput management from your side of the boundary. A worker pool sized against your rate limits is code you own: concurrency tuning, a token-bucket limiter, retry with jitter on 429s, backpressure when the limiter saturates, a dead-letter path for requests that exhaust their retries, and a dashboard so you can see all of it. With a batch endpoint you hand the provider 100,000 requests in a single call and none of that exists. Batch submissions draw on a separate enqueued-token allowance — a ceiling on how many tokens may sit in queued jobs at once — rather than the per-minute limits the pool contends with, so the pacing problem does not move to a different layer; it stops being your problem.

That is a real engineering saving and it is routinely undervalued next to the dollar figure. The worker pool in the review-scoring codebase was 340 lines and had been the subject of two production failures in a year: once when a limiter tuned against an old rate limit throttled the work to a third of its capacity for six weeks without anyone noticing, and once when a retry storm during a provider degradation turned a partial outage into a full one. Neither of those failure modes exists on the batch path, because the code that produced them does not exist. When you compare the two options, count that as part of the price difference rather than as a separate concern.

Everything else you might hope you are buying, you are not. The requests are not deduplicated for you, the outputs are not validated for you, and the job is not retried for you. Batch is a scheduling arrangement, and the scheduling is the entire product.

Where the money for the discount comes from is worth following, because the answer constrains everything after it. Serving a model requires accelerators that have to be bought and powered before anybody submits anything, and an idle accelerator-hour cannot be stored and sold later — it is simply gone. Demand across a day is not flat, so a provider sized for its busy hours has troughs in which that hardware is running and earning nothing. Work carrying no opinion about which hour it runs in can be moved into those troughs, and serving it there costs the provider less at the margin than serving it at a peak. The price cut is a share of that saving, handed back to the customer who supplied the flexibility that made it available.

Read the same mechanism in reverse and it settles what the window can and cannot mean. The discount is funded by the provider's freedom to run your work whenever its capacity is cheapest. A commitment about when your results arrive would remove that freedom, because honouring it means holding capacity in reserve against your job — and reserved capacity is exactly the idle capacity the discount is paid out of. A guaranteed delivery time and a 50% price cut are therefore not two features that happen not to be sold together. They are opposite ends of one trade. A tier that commits to a bounded completion time can certainly exist, and some providers sell one, but it has to hand back part of the discount to pay for the capacity it reserves — which is this same trade struck at a different point rather than an exception to it.

The window is an expiry, not a promise#

Both providers describe a 24-hour completion window, and both mean the same thing by it: requests still unstarted at the end of the window are marked expired, returned to you unprocessed, and not billed. OpenAI lets you request completion_window="24h" explicitly. Neither statement is a delivery time. The window is where the provider gives up, not where you get your data.

An expiry is the only kind of commitment the arrangement can support, and that follows directly from where the discount came from. A bound on how long the provider will hold unstarted work before handing it back is a promise to stop, not a promise to start, so it costs nothing in reserved capacity and a scheduler that is free to run your job at any hour can honour it. A bound on when your job completes cannot be honoured the same way, because meeting it means setting capacity aside for you. The 24-hour figure is therefore not a delivery estimate that the provider has hedged out of caution. It is the strongest statement this pricing model permits anyone to make.

The belief the expiry framing displaces is worth naming precisely, because it is the sentence that ends up in proposals: that batch results come back in a few hours. It is tempting for good reasons rather than careless ones. Anthropic's documentation really does say most batches finish considerably sooner than the window, that statement is published by the provider about the system you are about to use, and it is true. What it is not is a commitment, and the difference lies in what kind of fact it is. It describes how a queue has behaved under a load made up of other customers' submissions, which the provider does not control, and a description of past behaviour under uncontrolled load is a prior about tonight rather than a bound on it.

What sits between submission and completion is a queue whose depth you cannot see. Your job competes with everyone else's, and the load is not uniform — batch submissions cluster at the end of business days and around the top of the hour, because that is when scheduled work fires. The consequence is that the only honest way to answer "will this be back by 06:00" is to measure it on your own workload, submitted at the hour you would really submit, over enough runs to see a distribution rather than a number.

Nothing about that queue is observable from your side, and the gap is worth being exact about. You can read your own job's status and its per-request counts, and neither of those says how much work sits ahead of it, how much capacity the provider has free at that moment, or how many other customers submitted in the same minute. There is no published depth and no position number, and there is no reason to expect one, since what fills the queue is other people's traffic. That leaves exactly one instrument available: a queue you cannot model is a queue you can only sample.

Measuring it costs almost nothing, because the two timestamps you need come back on the job object:

batch = client.messages.batches.retrieve(job_id)
if batch.processing_status == "ended":
    db.execute(
        """
        update batch_runs
           set ended_at = %s,
               queue_seconds = extract(epoch from (%s::timestamptz - submitted_at))
         where provider_job_id = %s
        """,
        (batch.ended_at, batch.ended_at, job_id),
    )

That writes into the same batch_runs table the previous chapter created to hold the durable job id, so the measurement is a column on a row that already exists rather than a separate piece of instrumentation. Give it a few weeks and the question stops being arguable:

select date_part('hour', submitted_at)               as submit_hour,
       count(*)                                      as runs,
       round(percentile_cont(0.5)  within group (order by queue_seconds) / 3600.0, 1) as p50_hours,
       round(percentile_cont(0.9)  within group (order by queue_seconds) / 3600.0, 1) as p90_hours,
       round(max(queue_seconds) / 3600.0, 1)         as worst_hours
  from batch_runs
 where status = 'output_stored'
 group by 1
 order by 1;

On the review-scoring workload, 14 runs of 100,000 requests each submitted between 21:00 and 23:00 produced a median of 3.1 hours, a p90 of 9.3 hours — the ninetieth percentile, the time nine runs in ten came in under — and a single run that took 21.8 hours and finished with 4% of its requests expired. The same job shape submitted mid-morning had a median of 1.4 hours. That spread is the answer to "back the same day": usually yes, and the tail is long enough that a hard deadline cannot be planned against the median.

That one run repays being looked at rather than averaged away, because it is the window doing exactly what the first half of this section described. The job did not fail. It ran until the expiry, handed back the requests it had never started, and was not billed for them. A median of 3.1 hours and a night like that come out of the same queue within the same fortnight, and it is the second of the two that a deadline has to survive.

Because the thing being sampled is a queue filled by other customers' submissions, a sample describes the conditions it was taken under and nothing wider. What follows from that are three details, and they are consequences of the mechanism rather than general advice about running experiments carefully.

Three details decide whether that measurement means anything. Submit at the hour you would really submit, because a convenient afternoon test run and a 22:00 production run are sampling different queues. Use your real job size, since a 500-request probe and a 100,000-request job are not obviously scheduled the same way and you have no visibility into whether they are. And run it enough times to see a tail: five runs give you a median and no information at all about the ninetieth percentile, which is the number a deadline actually cares about. Ten to fifteen runs is where the shape of the distribution starts to stabilise, which for a nightly job means two weeks of ordinary operation rather than a dedicated experiment.

That last requirement is arithmetic rather than a rule of thumb. A percentile is an order statistic: you obtain the ninetieth by sorting the runs you have and reading off the value that one run in ten exceeds. In a sample of five there is no such position to read, because one-in-ten is not a countable event among five observations, so any p90 computed from them is an artefact of however the code interpolated rather than a measurement of anything. Fourteen runs place the ninetieth percentile between the second-worst and third-worst observations, which is still a coarse estimate and is at least an estimate of the quantity you asked for.

Note what the measurement does not tell you. It describes the provider's queue during the weeks you sampled it, on your job sizes, from your account. It is a prior, not a guarantee, and it will shift when the provider's load shifts or when a new model release changes who is submitting what. Re-measure on a schedule rather than treating a number from March as a property of the system.

Batch is a scheduling arrangement, not a slower model#

A single request inside a batch is served by the same weights at the same tokens-per-second as the same request sent synchronously; what you are waiting for is scheduling, not computation. That kills the intuition that batching is a quality trade-off. It is not, and outputs from a batched request are not systematically different from outputs from a live one.

The phrase "the same weights" is carrying that claim, so it is worth unpacking once. A batch request and a synchronous request are the same request object, naming the same model and served by the same weights running the same computation. Nothing on the batch path substitutes a smaller model, shortens the input or samples differently. The whole of the difference sits before the computation begins: a synchronous request starts being served as soon as it arrives, and a batched one waits in a queue first. Queue time is time in which no computation is happening at all, and nothing that happens during it can leave a mark on what the model eventually writes.

It is not a priority tier in reverse either. Submitting a smaller batch does not move you up the queue in any way you can rely on, and splitting one 100,000-request job into ten 10,000-request jobs mostly buys you ten status checks and ten result files. The observed variance across the 14 runs was not explained by job size.

The reason a smaller submission does not help is the opacity from earlier in the chapter, restated as a consequence. You are not at the front or the back of anything you can see, and the scheduler orders work against a pool you have no view of, so a smaller job is a smaller item in that pool rather than an earlier one. Splitting also has an effect the intuition gets backwards. Ten smaller jobs are ten draws from the same queue-time distribution, and you wait for the slowest of the ten — the same arithmetic that decided the four-hour question above, applied this time to a split you chose voluntarily.

And it is not a rate-limit workaround. If your synchronous path is throttled because your account limits are low, batch relieves that by moving the work to a different allowance, but the allowance is finite too, and a submission that exceeds your enqueued-token limit fails at submit time rather than degrading gracefully the way a worker pool does. The pool's behaviour under pressure is slow; the batch endpoint's is a rejection at the door.

That asymmetry comes from what each limit is a limit on. A per-minute rate limit governs work in flight, and a worker pool meeting one has somewhere to put the excess: it holds requests back and issues them a moment later, so pressure turns into latency. An enqueued-token allowance governs work that is waiting — the tokens sitting in jobs the provider has accepted and not yet finished — and a batch submission is a single indivisible call against it. It either fits inside the remaining allowance or it does not. There is no partial acceptance and nothing on your side to slow down, which is why the recovery for that rejection is to submit less at once rather than to submit more slowly.

Half price for a job whose completion time nobody will commit to is a good trade for most offline work and a bad one for anything with a wall clock attached to it, and the distinction is available for the cost of two timestamps per run. The discount is a fact you can put in a spreadsheet on day one. The completion time is a distribution with a long right tail that belongs to a queue you share with strangers, and the number that matters in it is not the median anyone quotes but the ninetieth percentile nobody measures. A team that has been recording queue_seconds for a fortnight can answer the deadline question in a minute; a team that has not is choosing between a rate card and a rumour.

The arithmetic above assumed something it should not have, though: that a job which comes back on time comes back correct. The review-scoring run that landed in 3.1 hours reported ended, downloaded cleanly, and contained 2,847 rows that were not scores at all. What those rows were, and why a job-level status check cannot see them, is the next chapter.