Classifying workloads for batch processing

The sorting rule most teams reach for is "is a human waiting", and it is wrong in a specific way: it asks who is watching rather than what happens when the result is late, and those are different questions with different answers for exactly the workloads that get misrouted. A batch endpoint — the asynchronous interface where a whole collection of model requests is submitted in one call, run on the provider's schedule and collected hours later as a results file — costs half as much as calling the same model one request at a time and publishes only an expiry window, not a completion time. Offline, scheduled and unwatched are therefore necessary conditions for using one, and nowhere near sufficient.

The rule is tempting because it is right most of the time and for a real reason. Latency requirements come from people, and most work with a person attached to it does need its answer quickly, so "is a human waiting" and "does this have a deadline" agree on the great majority of workloads anyone will classify. They part company on the jobs that inherit a deadline from something other than a watcher: a report that renders at a fixed hour, a release window, a contractual cutoff. In every one of those the answer to "is a human waiting" is no and the answer to "does the result lose its value at a fixed time" is yes, so the rule does not fail at random — it fails on precisely the set it is being used to decide. A rule that is correct on the easy cases and wrong on the hard ones is worse than no rule at all, because it has earned enough trust that nobody re-derives it.

This chapter replaces the rule with three readings that can be taken on any candidate workload before a line of code is written, runs seven real candidates through them, and then spends most of its length on the two cases where the classification is not the interesting part — a forty-million-document re-embedding job where the endpoint choice is worth two hundred dollars against weeks of engineering, and a nightly evaluation run where every surface reading says batch and the correct answer is synchronous.

The three axes#

Deadline is not "is this urgent" but "is there a wall-clock time after which the result loses its value, and what happens then". A backfill of archived data has no such time; a nightly aggregation feeding a morning report has one at 06:00; an evaluation run gating a release has one at 09:00 even though it began at 01:00 and nobody sat watching it. The axis has a second reading that matters just as much: if the deadline is missed, is there a fallback, and how many attempts fit before the deadline arrives? A deadline with a well-exercised synchronous fallback is soft in practice. A deadline with one shot and no alternative is the hardest kind there is, regardless of how relaxed the schedule looks.

What "how many attempts fit" means concretely is a division, and it is the step that turns the axis from a judgement into a reading. Take the time between when you would know the batch is late and the moment the result stops being worth having, and divide it by how long one attempt on the fallback path takes. A number comfortably above one means the deadline is soft: there is room to notice, decide and re-drive, and still land. A number at or below one means the first attempt is the only attempt, and no amount of alerting improves it, because the alert arrives inside a window too small to act in. The reason this is a reading rather than a guess is that both quantities are measurable before anything ships — the schedule fixes the first and a load test of the fallback fixes the second.

Retry granularity asks whether a single item can be re-driven on its own and whether re-driving it is safe. That means two things: the work has to be decomposable to the item, which is what the per-item reconciliation table from chapter 3 assumes — one row per submitted custom_id, carrying a terminal state — and the write has to be idempotent, because at scale some items will be written twice. A workload where re-running one row is a no-op if it already succeeded fits the batch machinery perfectly. A workload where the unit of correctness is the whole run — an aggregate, a ranking, a report — cannot use item-level retry no matter how the requests are shaped, because 99.8% of an aggregate is not 99.8% of an answer.

That last sentence carries the whole axis, so it is worth taking apart. An aggregate is a value computed over a set of items — a mean score, a pass rate, a ranking — and a function evaluated on a set that is missing some of its members does not return a slightly wrong answer of the same kind. It returns the correct value of a different quantity: the pass rate of the comparisons that came back, which is not the pass rate of the suite. Nothing downstream can tell the two apart, because they have the same type, arrive in the same field and look equally plausible. Item-level retry is safe exactly when the unit of correctness and the unit of the request are the same object, and an aggregate is the case where they are not.

Ordering covers two distinct constraints that behave the same way here. One is sequencing: item N's input depends on item N−1's output, which a batch endpoint cannot express at all, since every request in a job is independent and served in whatever order the scheduler prefers. The other is consumption: whether downstream can read partial results as they land or has to wait for the complete set. A backfill nobody reads until it finishes is the friendly case; a table queried continuously while being rewritten is not.

That a batch endpoint cannot express sequencing is a consequence of when the submission happens, not a gap somebody forgot to fill. Every request in a job is handed over in a single call, before any of them has run. A request whose input is another request's output would therefore have to carry a value that does not exist at the moment it is written down, and no field can hold a value that has not been produced. Ordering the file differently does not help, because the problem is not which request runs first but that the second one's text is already fixed. The scheduler's freedom to run the requests in whatever order suits its capacity depends on exactly this: none of them can refer to another. That freedom is part of what the discount is paying for, so sequencing is not a feature a batch endpoint happens to lack — it is the thing that was sold.

Run four candidate workloads through the axes and the ranking is unambiguous:

The backfill wins on all three axes at once, and that combination — no deadline, per-item idempotent retry, no ordering constraint — is what a batch endpoint is built for. Notice that two of the losing rows lose on the deadline axis and one loses on ordering, and that none of them lose because of volume. Volume is what makes batch attractive; it is not what makes it appropriate.

Three more candidates arrived in one sprint at the same company, and taking the readings on them is where the axes start to earn their keep. The first involves a term worth defining: an embedding is the list of numbers a model produces to stand for a piece of text, and a similarity search finds related documents by comparing those numbers, so re-embedding forty million documents means recomputing that list for every one of them after a change that made the existing lists unusable.

Where the re-embedding job is actually hard#

Forty million documents, re-embedded after an index migration, is the workload that looks like the biggest engineering problem and has the smallest endpoint decision inside it. At roughly 500 tokens per document and typical embedding pricing near $0.02 per million tokens, the whole job is about $400 synchronously and about $200 on the batch endpoint. Two hundred dollars is the entire prize, on a project that will take an engineer several weeks. Whatever makes this job hard, it is not the endpoint, and a design review that spends its time on batch-versus-sync has spent it on the cheapest decision available.

Both figures are one multiplication against a rate quoted per million tokens. Forty million documents at roughly 500 tokens each is twenty thousand million tokens, which is twenty thousand of the million-token units the rate card prices in, and twenty thousand of them at two hundredths of a dollar apiece is the $400. The batch column is the identical product against a rate halved with no minimum volume and no tier, and halving every term of a sum halves the sum, so the whole bill moves to about $200 rather than some part of it.

What is hard is that 40 million item-level writes will be interrupted, resumed, and partially repeated, and the index has to be correct at every moment in between. That is a schema problem:

create table embeddings (
    doc_id         bigint  not null,
    chunk_ix       int     not null,
    model_version  text    not null,
    content_hash   text    not null,
    vector         vector(1536) not null,
    written_at     timestamptz not null default now(),
    primary key (doc_id, chunk_ix, model_version)
);

insert into embeddings (doc_id, chunk_ix, model_version, content_hash, vector)
values (%(doc_id)s, %(chunk_ix)s, %(model_version)s, %(content_hash)s, %(vector)s)
on conflict (doc_id, chunk_ix, model_version) do update
   set vector       = excluded.vector,
       content_hash = excluded.content_hash,
       written_at   = now()
 where embeddings.content_hash is distinct from excluded.content_hash;

model_version in the primary key is what lets both embedding generations coexist while the backfill runs, so queries keep serving the old vectors until the new set is complete rather than reading a table that is half one model and half another. The mechanism is worth one step down: a primary key is the identity of a row, so putting the model generation inside it means a new vector for a document that already has one is a different row rather than an overwrite of the existing one. Two generations can then be present at the same time, and the cutover becomes a change to which model_version the query names rather than a race between readers and a rewrite.

content_hash makes a repeated write a genuine no-op, which matters because the retrieval step in chapter 1 was deliberately made safe to re-invoke and the retries in chapter 3 deliberately re-send items. The where clause is where that happens: a retry that re-sends an item it has already written finds the two hashes equal, the update is skipped, and written_at does not move, so a repeated write leaves no trace at all. It is written as is distinct from rather than <> because comparing anything with <> against a null yields null rather than true, and a row whose hash had never been recorded would then fail the test and never be updated — which is the one case the clause exists to handle.

The cutover is gated on the coverage metric from chapter 3 — succeeded items divided by items submitted — reaching 1.0 for the new model_version, not on the last batch job reporting ended. A mixed index is worse than an old one, because its similarity scores are incomparable across generations in a way that produces plausible, wrong results rather than obvious failures.

That claim deserves its mechanism, because it is the reason the coverage metric gates the cutover rather than merely reporting on it. A similarity search ranks documents by comparing the query's list of numbers against each stored list, and that comparison means something only when both lists were produced by the same model, since a model settles for itself what each position in the list stands for. Two generations put the same document at two different points, so a score computed across generations is not a weaker measurement of similarity — it is a measurement of nothing, arrived at by arithmetic that still succeeds and still returns a number in the usual range. A ranking assembled from a mixture is therefore ordered partly by which model happened to write each vector. An old index is consistently stale, which shows up as results that are recognisably out of date; a mixed index is inconsistently wrong, which shows up as nothing at all.

Small jobs with expensive consequences#

The general form of that decision is worth extracting, because it recurs: the value of the discount scales with token volume, while the cost of a late result scales with what depends on it, and the two are entirely unrelated. Small jobs with expensive consequences are the worst possible fit for a batch endpoint, and they are also the jobs that look most obviously batch-shaped from a distance, because they are offline, scheduled and unwatched.

What breaks the link between them is that each is set by something the other cannot reach. A discount is a fraction of a token bill, so its value can only move with how many tokens the job puts through the model. Lateness costs whatever is waiting on the answer — a release window, a contractual report, a team's morning — and none of those has any dependence on a token count. A job can therefore be tiny and consequential, or enormous and inconsequential, and any sorting rule that reads volume is reading whichever of the two happens not to be deciding. That is also why the eval suite is a worse fit than the forty-million-document backfill despite being smaller than it by four orders of magnitude: the backfill has nothing waiting on it, and the suite has a release.

The moderation workload fails faster and for a simpler reason: publication blocks on the result, so an hours-long window is not a risk to manage but a contradiction. There is a legitimate hybrid, though, and it is worth naming so the axes do not read as a ban. Run a fast synchronous classifier in the request path to make the publish decision, then submit the ambiguous tail — the items the classifier scored near its threshold — to a batch endpoint for a deeper overnight review that can retract or confirm. The blocking decision keeps its latency budget and the expensive second opinion gets the discount, because they are two different workloads that happen to share an input.

What makes that split legitimate rather than a way of smuggling the deadline back in is that the two halves take the axes differently. The blocking decision has a deadline of seconds, one attempt and no fallback, so it stays synchronous. The second opinion has no deadline at all, because the item is already published or already blocked by the time it runs, and its result is a correction rather than a gate. Splitting a workload is only available when a split like that exists in the work itself — when part of it genuinely stops being urgent once the urgent part has answered. A job that cannot be divided that way does not become dividable by being routed twice.

Intermediate service tiers, and work that need not happen#

The axes say nothing about whether the work should happen at all, which is usually the larger saving. A backfill that re-embeds 40 million documents including the 30% nobody has queried in two years is a job that is 30% too big before any endpoint is chosen, and no discount recovers that.

They also say nothing about the state of the art, which is moving in a direction that will blunt them. Providers are beginning to offer intermediate tiers between batch and standard — smaller discounts in exchange for a bounded, sometimes contractual, completion time — and if a tier exists that guarantees results within, say, two hours at 70% of list price, the eval suite becomes a straightforward yes rather than a close call. The axes stay valid; what changes is that the deadline axis stops being binary and starts having a middle. Re-check your providers' tier list before treating any of these classifications as settled, because a rate card is a cheaper thing to change than a pipeline.

Deadline, retry granularity and ordering are three readings taken in about ten minutes, and between them they route every workload in this chapter correctly, including the two that the usual rule sends the wrong way. What they are really testing is whether a job can tolerate an answer arriving at an hour the provider chooses, whether damage inside that answer can be repaired one item at a time, and whether anything downstream cares about the order or the completeness of what lands. A workload that clears all three is one a batch endpoint was designed for, and the discount on it is free money. A workload that fails one of them does not become a better fit for having a large token bill.

The way to make all of this concrete is to run one. Take the workload in front of you that scores cleanly on all three axes, submit a real batch of a few thousand rows, and build the pieces in the order this topic built them: persist the job id, store the raw output, reconcile per item against the ids you sent, measure the queue-to-completion spread over a couple of weeks, and only then decide what your cutoff and fallback should be. The mechanisms are individually small. What makes them worth having is that they compose into a pipeline whose worst night is a slightly expensive one rather than a silent, week-long corruption.