A queue transports work; it never remembers where you were

The contract a queue offers is narrow, complete, and about something other than your run. That is not a criticism, and reading it as one leads teams to configure their way towards a guarantee the product does not sell. What it means in practice is that every question of the form "what is this unit of work doing" has to be answered by something you built, because the queue's vocabulary has no word for a unit of work that spans more than one message.

Naming the mismatch precisely is what makes the rest of this chapter mechanical. Ask of any queue feature whether it is expressible as a statement about a message. Retention, visibility, redelivery, dead-lettering and in-flight limits all are. "Which step is this order on", "wake this run on Thursday" and "did this run already charge the card" all are not, and no configuration turns one kind into the other.

The console that could not answer#

The question in an incident channel was "where is PO-2026-0193 right now", and the on-call engineer opened the SQS console first, because that is where the work lives. The console told her the queue had 4 messages available, 61 in flight, an oldest message age of 1,240 seconds, and 14 messages in the dead-letter queue. Not one of those numbers is about PO-2026-0193.

Those four numbers are not an unlucky selection; they are close to the complete set of facts a queue holds. A broker indexes messages by the order they arrived and by whether they are currently visible, because those are the two facts it needs to do its job. It does not index by anything inside a body, since the body is opaque bytes to the broker. So there is no lookup by order reference, and there is no way to read a body in order to search it either: receiving a message is the only way to see its contents, and receiving it starts its visibility timer and takes it away from whichever worker might have been about to get it. Searching a queue changes it.

She could not tell from the queue whether that order had been credit-checked, whether it was waiting for an approver, or whether a worker was holding it at that moment, because the queue does not model runs. It models messages, and a message is a delivery attempt with a body. The answer came from the run_state table in about four seconds, which is the point. Every fact she needed had been built by hand over the previous five chapters, and none of it came from the transport.

What the queue actually promises#

The contract is narrow and it is genuinely good. A queue stores a message durably across multiple availability zones, delivers it at least once, hides it from other consumers for a visibility timeout while one worker works on it, redelivers it if the worker does not acknowledge, moves it to a dead-letter queue, abbreviated DLQ, after a configured number of receives, and absorbs a burst so producers do not have to slow down when consumers do. Amazon SQS puts numbers on all of that: retention up to 14 days, visibility timeout up to 12 hours, a delivery delay up to 900 seconds, 120,000 in-flight messages on a standard queue, and roughly $0.40 per million requests. Those are real guarantees and they cover the transport problem completely.

They are also the entire list. Nothing in it mentions the word "run". The queue has no concept that message A and message B belong to the same unit of work, no place to record which step that unit reached, no way to answer a question about a specific business entity, and no notion of a wait that lasts three days. Every one of those is a property of your application, and the previous five chapters are the bill for them.

Read the middle column rather than the rows. Where it says something, the capability is about a message; where it says "Nothing", the capability is about a run. The one row where both columns have content is single ownership, and that is the row where the two units nearly coincide — a visibility timeout is a claim on a message for a while, which is almost a lease and is not one, because it lapses on a schedule the broker owns and carries no counter a stale worker's write can be tested against.

The two capabilities the table leaves to you#

The two rows with no chapter behind them are the ones that catch teams last. Fan-out is easy on a queue, since publishing 240 messages and having eight consumers work them is exactly what queues are for. The join is not, and the reason is worth deriving rather than asserting, because it explains why the bugs are always the same three.

Publishing 240 messages requires no shared state at all: each publish is independent and the consumers never have to know about each other. Knowing that all 240 have finished requires exactly the opposite — a piece of mutable state that every child updates and that somebody reads at the instant it reaches zero. That is shared mutable state with concurrent writers, which is the thing the queue's model removed, so it has to come back as a counter row updated transactionally by each child and a condition that wakes the parent.

Each of the three well-known bugs is one property of the queue meeting that counter. Delivery is at-least-once, so a redelivered child decrements twice unless the decrement is keyed on the child's identity — that is double-decrement on redelivery. Two children can observe the counter reaching zero if the read and the wake are not in one transaction, which wakes the parent twice. And a child that fails permanently never decrements, so the counter sits one short forever and the parent waits for something that is not coming, which needs a timeout the queue does not supply either.

The versioning row fails for a simpler reason and is easier to miss because nothing goes wrong until a deploy. A message carries a body and no notion of which version of the code produced it, so a consumer receiving one cannot tell whether it was published by the code currently deployed or by the code that was running an hour ago. Change the step sequence and messages already in flight are addressed to a pipeline that no longer exists in that shape. The only remedies are to put a version field in the body yourself and branch on it, or to drain the queue before deploying — and draining is unavailable exactly when runs are long, which is when the problem matters.

A log-based broker like Kafka is sometimes offered as the answer here, on the grounds that it does persist a position. The position it persists is the consumer's offset in a partition, one of the ordered append-only slices a named stream of messages is split across, which is a statement about how far a consumer has read, and it is not a statement about how far any individual run has progressed. Two runs interleaved in the same partition share one offset; committing it means both are considered read, so a consumer that has finished one run and is halfway through another cannot express that with an offset. Log compaction, which keeps the most recent message for each key indefinitely and discards the older ones, together with long retention, gives you a durable record of the events, which is genuinely useful for reconstructing history, and reconstructing a run's position from a topic means reading and folding its events yourself — which is a checkpoint built from a stream instead of a table, with the same code around it.

The pattern that looks like it solves this#

The common design is step chaining: each step, when it finishes, publishes the next step's message, carrying the accumulated state in the body.

{
  "run_id": "run_8f3c1a",
  "next_step": "charge",
  "state": {
    "order_ref": "PO-2026-0041",
    "supplier_id": "sup_881",
    "total_cents": 1420000,
    "credit_score": 712
  }
}

This is a real pattern and it works. For a three-step pipeline with no human in it and no writes worth deduplicating, it is the correct amount of machinery and reaching for anything heavier is waste. What makes it work there is that the run's position and a message boundary are the same thing: the run is between steps exactly when a message is in the queue, so the queue's unit and the run's unit coincide and nothing is missing. Its limits are the four ways that coincidence breaks, and they are worth knowing before the pipeline grows to nine steps.

The message becomes the checkpoint, so the state travels rather than resides. Nothing queryable holds the current position, which is why the console could not answer the on-call's question; the answer was inside a message body that no API lets you read without consuming it. Recovering visibility means writing the state to a table as well, at which point the message is a wake-up signal and the table is the checkpoint, and you have built chapter 1 anyway.

The payload has a ceiling. SQS caps a message at 256 KB, and an agent's accumulated state — a transcript, 240 priced line items, a supplier record — passes that around step four, so the body becomes a pointer into S3, Amazon's object storage, and the state is back in a store you manage. That is chapter 1's two-store problem arriving through the payload rather than through the design.

Redrive, the operation that moves messages out of the dead-letter queue and back onto the main one, replays a step whose predecessors already happened, which is chapter 4's problem arriving through a different door. This one is worth flagging as a trap rather than a limitation, because redrive is the recovery button and an operator pressing it believes they are recovering. A message that lands in the DLQ and is redriven a day later re-executes charge with whatever state it was carrying, and unless that state includes an effect key and the ledger is consulted, the redrive is a duplicate charge with an operator's name on it. The metric everyone watches — DLQ depth — goes to zero exactly as intended, while the count of duplicated external effects goes up, and only one of those two has a dashboard.

And the chain has no memory of itself. There is no per-run history, so an audit question about what the run did and when has to be reconstructed from application logs, and a change to the step sequence applies to messages already in flight with no way to distinguish old ones from new.

The boundary, stated for an interview#

The shallow version of this comparison is that an engine is a queue with better ergonomics, and it is wrong in a specific way. Ergonomics is a consequence rather than the product. The distinguishing property is that a durable engine persists execution position — the place in your program where the run had reached, including the local state that got it there — so a worker that no longer exists can be replaced by one that has never seen the run and the run continues from where it was, not from the last message boundary somebody remembered to publish.

A queue persists a message. If your run is nine steps and you want position, you publish nine messages and you write the state, the lease, the boundary, the keys, the ledger, the timers and the pause table around them. That is not an ergonomic difference; it is a different guarantee, and the code in chapters 1 through 5 is what the difference weighs.

The rest of the boundary follows from that one property, which is why the list of things an engine offers looks arbitrary until you have it. Durable timers exist because a run's position can be "waiting until Thursday" rather than "not currently a message". Replay semantics exist because position implies a rule for what happens on re-execution. Coordination exists because two runs with positions can wait on each other. Versioning exists because a run holding a position in a program is holding a position in a specific version of that program, which is a problem a stateless consumer never has.

What the queue is better at#

Setting the boundary honestly requires saying where the queue wins, because a durable engine is not a superset of a queue in the dimensions that matter operationally. A queue is one managed dependency with a two-line client, no new programming model, no constraint on how your code is written, and a cost that rounds to nothing: 17,000 messages a day on SQS is under a euro a month. It scales horizontally by adding consumers, it absorbs a supplier's three-hour outage by simply growing, and every engineer on the team already understands it. Nothing about adopting it changes how the agent code is written.

That last property is the one that does not appear on any comparison table and matters most in practice. Adopting a queue imposes no rule on the inside of your functions, so it cannot be violated by a code change, cannot fail at replay time, and cannot stall a run because somebody deployed. The next chapter's mechanism can do all three, and that is the trade rather than a defect in either.

Those properties are also why the queue does not disappear when an engine arrives. Durable engines run task queues internally and most deployments keep an external queue in front for ingestion, because the engine is the wrong place to absorb a burst of 50,000 webhook deliveries. The two compose: the queue moves work to workers, and something else remembers what each unit of work was doing.

Teams that only need transport and retry buy a real cost for a guarantee they will not use, and chapter 8 makes that argument in full. What remains to establish first is what the guarantee actually is, because what an engine does with position is not what chapters 1 through 5 did with it. The hand-built system stores a cursor and re-enters a function at a named step, which is a design you could reasonably assume an engine implements more neatly. It does something quite different, and the difference explains both the strength of the guarantee and the constraints it imposes on your code.