Batch and async job endpoints

Read these first

• Concurrency and throughput

What you will know

• Submit a batch job and collect the results

  Verify authentication, request-file formatting, polling or completion handling, and durable storage of the output rather than relying on an interactive session.

• Measure the cost and end-to-end latency against synchronous calls on the same workload

  Compare per-token or per-request spend and queue-to-completion time; advertised discounts do not predict the completion time of a busy service.

• Classify three candidate workloads for batch processing using explicit deadline, retry, and ordering requirements

  Backfills and evaluation runs usually qualify, but user-facing enrichment or workflows with strict sequencing may not tolerate an hours-long completion window.

• Handle partial failure within a batch

  A successful job can contain failed rows, malformed outputs, or expired requests, so retries and reconciliation must operate at item granularity.

• Run a batch under a stated deadline and implement the fallback when results miss it

  A timeout policy might switch to synchronous processing, defer the work, or alert an operator; the discount is not useful if downstream data is needed sooner.

Chapters

1. 1

   The batch job round trip

   A batch job keeps running on the provider's infrastructure long after the process that submitted it has died, so the job identifier returned at submit time is the only durable handle on work that is already being billed. This chapter walks the whole round trip: request-file formatting, the custom id that joins a result back to the row it came from, the credential and the enqueued-token limit that govern a submission, the database write that has to happen the instant the id exists, and a drain step that stores the raw output before anything parses it.

   Read chapter
2. 2

   Batch pricing and the completion window

   A batch endpoint sells two things a rate-limit-aware worker pool cannot: tokens at half price, and the removal of throughput management from your side of the boundary. It does not sell a delivery time — the published 24-hour window is where unstarted requests expire, not where yours finish. This chapter works through the pricing arithmetic, shows how to measure queue-to-completion on your own workload at the hour you would really submit, and sets a measured distribution against a hard deadline.

   Read chapter
3. 3

   Partial failure within a batch

   Failure inside a batch job is reported per item, so the job-level status is green while a few percent of the rows are provider errors, requests that expired unstarted, or model outputs that parse but fail your schema. This chapter separates those three kinds of damage, classifies every line of the output file into a terminal state before extracting any payload, and builds a reconciliation table keyed by the custom ids you submitted rather than the ones that came back — which is what makes retries item-level and coverage measurable.

   Read chapter
4. 4

   Cancelling a batch job

   Stopping your poller leaves the provider's job running and billing, so abandoning a batch is an API call rather than a client-side decision. A cancelled job still produces an output file for the requests that already completed, and diffing those custom ids against the ones you submitted gives the outstanding set — which is the difference between re-driving eighty-eight thousand rows and re-driving three hundred thousand. This chapter covers what cancel does, how long it takes to settle, and what the partial results are worth.

   Read chapter
5. 5

   Batch deadlines and fallback policy

   A timeout on a batch poller that only raises an exception converts a slow run into a fast failure, which helps nobody when the thing downstream needs rows rather than an alert. The cutoff has to be the deadline minus however long the fallback needs, computed from the provider's live per-request counts rather than fixed in a constant. This chapter derives the moving cutoff, prices what firing it costs at every level of batch completion, and identifies the slip rate at which the whole design should be relabelled.

   Read chapter
6. 6

   Classifying workloads for batch processing

   \"Is a human waiting\" sorts workloads by who is watching rather than by what happens when results are late, which is why it routes a nightly eval that gates a morning rollout onto a batch endpoint. Three axes do better: whether a wall-clock deadline exists and what backs it up, whether work decomposes to idempotent per-item retries, and whether anything downstream needs ordering or completeness. This chapter runs seven candidate workloads through them and shows where the real engineering difficulty sits.

   Read chapter