Derived work belongs behind a buffer, not on the request path

At 09:14 a collector deployment rolled in the observability namespace and the new pods came up without a working DNS entry for their upstream. Nothing in the LLM service changed that morning. By 09:17 its p95 had gone from 940 ms to 5.9 s, by 09:22 the gunicorn workers were saturated and the gateway was returning 502s on about a third of requests, and the incident channel spent eleven minutes looking for a provider outage that did not exist. The cause was four lines in the response handler:

finally:
    span.set_attribute("llm.completion_tokens", usage.completion_tokens)
    span.end()
    exporter.export([span])        # blocking, 5 s timeout, no queue
    judge.score_async(request_id)  # "async" only in the name of the method

The exporter was configured with a simple span processor rather than a batching one, so every request made an outbound gRPC call and waited for it. A span is the record one unit of work leaves behind in a distributed trace — its name, its start and end times, and a bag of attributes — and exporting it means shipping that record to the backend where a human can read it afterwards. With DNS failing, each call burned its full five-second timeout before giving up. The judge call, despite the method name, awaited an HTTP POST to an internal scoring service, adding another 400 ms in the good case; a judge here is a second model call that reads a completion and rates it against a written rubric, which is how a system gets a quality number on traffic that nobody labelled by hand. A monitoring system took down the thing it was monitoring, and it did so through a mechanism that has nothing to do with LLMs: work that nobody was waiting for had been placed where the user was waiting.

The test is dependency, not cost#

The instinct after an incident like that is to move slow things off the request path, and the instinct is close enough to be dangerous. Latency is a symptom, and cost is a budget concern; neither tells you where a piece of work belongs. The question that decides it is whether anything downstream actually reads the result before the response is complete or before the next request runs.

A 12 ms Redis write belongs on the request path if the next turn reads it, and a 3 ms HTTP POST belongs off it if the only consumer is a dashboard a human looks at tomorrow. The Redis write is cheap and synchronous because the system's correctness depends on it having happened; the POST is cheap and asynchronous because nothing in the system's behaviour changes if it happens ninety seconds late, or if it is retried four times, or if it is dropped after exhausting retries.

Applied to the layers from chapter 1, the work in a chat request sorts cleanly.

Three rows in that table are the ones teams most often get wrong in the expensive direction, and they are the three that the incident above hit. Trace export, evaluation scoring and the priced cost record are all computed from a request that has already finished. None of them can change a token that was already streamed, and none of them is read by the next request. Their correct failure mode is a gap in a chart.

The row that is most often got wrong in the other direction is the usage receipt. It is tempting to lump it in with the cost record, because both have the word "usage" near them, but they are different objects. The receipt is a durable fact about what happened — this request id, this model, 1,840 prompt tokens, 612 completion tokens, committed alongside the assistant turn. The cost record is a pricing computation over receipts, and pricing changes, gets recomputed, and is aggregated per tenant per hour. The computation is derived and can run late. The fact it is derived from cannot be reconstructed from anywhere else once the process exits, so it is committed on the request path.

A buffer is not a background task#

Moving work off the path is an architectural change, and the version of it that most codebases actually ship is not. Firing asyncio.create_task(...) or go func() { ... }() in the response handler removes the latency and leaves everything else broken: there is no durability, so a pod rotation during a deploy drops every in-flight item; there is no backpressure, so a slow consumer becomes unbounded memory growth in the web process; there is no retry, so a transient 500 from the scoring service silently loses the record; and there is no visibility, because the failure happens in a task nobody is awaiting.

What the boundary actually needs is a queue with durability the producer can rely on, which for the events that matter means the producer must not be able to commit its work and lose the event. The transactional outbox is the standard way to get that without a distributed transaction: the event row is inserted in the same database transaction as the turn and the receipt, and a separate process reads it and publishes.

CREATE TABLE outbox (
  id          BIGSERIAL PRIMARY KEY,
  request_id  UUID        NOT NULL,
  event_type  TEXT        NOT NULL,   -- 'call_completed'
  payload     JSONB       NOT NULL,
  created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at TIMESTAMPTZ
);
CREATE INDEX outbox_unpublished ON outbox (id) WHERE published_at IS NULL;

async def finish_turn(ctx: RequestContext, turn: Turn, usage: Usage) -> None:
    async with db.transaction():                       # one transaction
        await db.insert_turn(turn)
        await db.insert_usage_receipt(ctx.request_id, usage)
        await db.insert_outbox(
            request_id=ctx.request_id,
            event_type="call_completed",
            payload={
                "tenant_id": ctx.tenant_id,
                "prompt_template": turn.prompt_template_id,
                "prompt_version": turn.prompt_version,
                "model": usage.model,
                "prompt_tokens": usage.prompt_tokens,
                "completion_tokens": usage.completion_tokens,
                "retrieved_chunk_ids": turn.retrieved_chunk_ids,
                "latency_ms": turn.latency_ms,
            },
        )
    # response returns here; nothing below is awaited by the caller

The relay is a loop that reads unpublished rows in id order, publishes them to the broker, and marks them published. It is allowed to publish the same event twice — a relay that crashes after publishing and before marking will — so every consumer of these events is keyed on request_id and treats a repeat as an update. The trace exporter, the scorer and the cost rollup all become consumers of that one event, which means the request handler stops knowing they exist.

The direct cost of the outbox is one extra insert inside a transaction that was already open, which measured about 0.8 ms on the service that inspired the incident above, against the 5.4 s the inline exporter was contributing at its worst. The indirect cost is a relay process and a broker to operate, and if you do not already run a broker, a table plus a SELECT ... FOR UPDATE SKIP LOCKED poller at one-second intervals is a legitimate implementation that nobody needs to learn a new system for.

Streaming moves the line without erasing it#

A streaming response complicates the picture in one specific way: the response is finished from the user's point of view long before the handler is, and the temptation is to treat everything after the last delta as post-request work. The turn commit is the case where that reasoning fails. If the assistant turn is written by a consumer of the event rather than by the handler, there is a window — a few hundred milliseconds under normal load, several seconds when the relay is behind — during which the user has read an answer that the conversation history does not contain. A user who sends a follow-up inside that window, which is exactly what an engaged user does, gets a prompt composed from a history missing the turn they are replying to, and the model answers a question that appears to come from nowhere.

The rule that survives streaming is that the commit happens before the stream closes, not after the last token. In practice the handler writes the turn, the receipt and the outbox row in one transaction, and only then sends the terminating data: [DONE] event, which costs about 6 ms on the p50 path and removes the window entirely. Everything computed from that committed state stays on the far side of the line.

The other thing streaming changes is the shape of the deadline. The gateway's timeout was set against a complete response, and a stream that has delivered tokens for 40 seconds is healthy while a unary call at 40 seconds is not, so the deadline the orchestrator enforces on a streaming call is an inter-token deadline — no delta, meaning no further fragment of the answer arriving on the stream, for 10 seconds is a failure — plus an absolute ceiling. Getting this wrong produces the specific incident where a healthy long answer is killed at 30 seconds by a timeout designed for a JSON endpoint.

Where the boundary stops helping#

Asynchronous does not mean unimportant, and this is where teams over-correct. Once trace export moves behind a buffer, the buffer needs its own alerting, because a relay that has been stopped for two days produces exactly the same green dashboards as a healthy system. The useful signal is queue depth and oldest-unpublished age rather than error rate — an alert at max(now() - created_at) > 5 minutes on unpublished outbox rows catches a stalled relay in a way that no per-request metric can.

The boundary also does not make work optional. A cost record that is ninety seconds late is fine; a cost record that is silently dropped once every thousand requests is a billing defect, and it will be discovered by a customer comparing your invoice to their own token counting. Behind the buffer, the delivery guarantee has to be at-least-once — every event is delivered, and some are delivered more than once — paired with idempotent consumers, meaning consumers for which handling the same event twice leaves the same result as handling it once. Reaching for at-most-once, where an event is delivered no more than once and may be dropped, because it is simpler moves the failure from the request path into the ledger.

The boundary also costs something in debuggability, and the cost is real enough to plan for. Before the change, a single trace contained the whole story of a request; afterwards it ends when the response does, and the scoring, the export and the cost record happen in three other processes with three other traces. Recovering the story means every consumer propagating the original request_id as a trace attribute and, where the tracing backend supports it, a span link back to the producing span. Without that propagation the first incident that involves the scoring pipeline is investigated by grepping logs for a UUID, which works and takes an hour longer than it should.

And there is a class of work that looks derived and is not. Anything the user is shown as part of the same interaction — a safety filter's verdict, a citation list rendered under the answer, a structured-output validation that decides whether to retry with a repair prompt — is read by the response, so it stays synchronous no matter how much it resembles evaluation. The tell is the direction of the dependency rather than the name of the component: a judge model scoring a completion for a weekly report is derived, and the same judge model gating whether the completion is shown is on the path.

What the boundary still does not tell you#

Sorting work by dependency answers when something runs. It says nothing about who is allowed to do it more than once. The failure that produced the incident review in the next chapter had every piece of work on the correct side of this boundary: retrieval was synchronous, the turn was committed before the response closed, tracing was buffered. The request still produced a duplicated answer, a corrupted conversation and a double charge, because two different layers each retried the same operation while holding a different idea of what that operation was.