Reference architecture

Read these first

• Multi-provider abstraction

What you will know

• Draw the production architecture from gateway to observability

  Include request and data flows, synchronous versus asynchronous boundaries, and ownership at each handoff so operational gaps are visible.

• Trace a request through the architecture and assign ownership for each operation

  Retries, streaming, state mutation, and provider failover often cross boundaries where unclear ownership creates duplicate work or inconsistent state.

• Identify the missing layer in an existing deployment

  A system can serve requests while lacking durable traces, quality signals, or evaluation data needed to detect silent regressions.

• Compare this architecture with a conventional service architecture and record the added failure modes and controls

  Nondeterminism, prompt and model changes, provider drift, and retrieval quality add controls that ordinary request monitoring cannot provide.

• Defend a simpler architecture where the full one is overkill

  Every extra store or control plane adds latency, failure modes, and operational cost, so the design should match traffic, risk, and change rate.

Chapters

1. 1

   The five layers and the decision each one owns

   Gateway, orchestration service, provider clients, the vector and state stores, and the evaluation plane sitting across all of them. Each layer exists because some decision has to be made in exactly one place: who the caller is, what the prompt says, which provider answers, what survives the request, and whether the answer was any good. Draw it once with request and data flows marked, and the operational gaps become visible before an incident finds them.

   Read chapter
2. 2

   Derived work belongs behind a buffer, not on the request path

   Trace export, evaluation scoring and the cost record are all computed from a request that has already finished, so none of them can change what the user sees or what the next turn reads. Move them behind a queue and their failure degrades visibility rather than the product. The test for the synchronous boundary is not cost or latency, it is whether anything downstream actually depends on the result.

   Read chapter
3. 3

   Retry belongs above the layer that owns the visible effect

   A provider client can safely retry a call whose effects never escaped it. Once 400 tokens have reached the browser, or a partial turn has been appended to Postgres, only the orchestrator knows what was shown and can choose to resume, restart or fail. The shared-client argument is right about duplication and wrong about scope, and the answer differs between a unary JSON extraction and a streaming chat turn.

   Read chapter
4. 4

   One duplicate is free, the other corrupts durable state

   A mobile client times out at 30 seconds and re-POSTs a request that had in fact succeeded, leaving the user with two identical answers and two charges. The expensive-looking duplicate, the second generation, is not the one that did the damage. Enforce the idempotency key at the orchestrator, on a client-supplied request id, written in the same transaction as the turn and the usage record.

   Read chapter
5. 5

   A green dashboard is compatible with a wrong answer

   A platform with distributed tracing, per-service SLOs, canary deploys, circuit breaking and cost allocation already covers most of what an LLM feature needs. What none of it can do is notice that the same input now returns a worse answer with no error, no latency change and no deploy. Nondeterminism, prompt and model version drift, and retrieval quality are the failure modes ordinary request monitoring is structurally blind to.

   Read chapter
6. 6

   Per-call records are what turn complaints into a metric

   Support has quietly kept a spreadsheet of bad answers for three weeks while every dashboard stayed green. The missing plane records prompt, retrieved chunks, model version and completion for each call, joined to outcome labels and sliced by document type, tenant and request class. An unversioned, unsliced judge score averaged over mixed traffic imitates this plane and yields a number nobody can act on.

   Read chapter
7. 7

   The cheapest component is the one you cannot reconstruct later

   At 200 requests a day with one prompt, one provider and a human reading every output, the full architecture is a liability: each extra store and control plane adds latency, failure modes and operating cost. Keep per-call logging of prompt, model version and completion, since that is the one thing no one can recover afterwards. Volume is the wrong variable — risk, change rate and who consumes the output decide it.

   Read chapter