The five layers and the decision each one owns

Six engineers spent three hours on a design review scheduled for one, and the diagram on the screen was the reason. It had four boxes — web, api, llm-service, openai — joined by three arrows, and it was an accurate picture of what was deployed. It could not answer any of the four questions the review had to settle before the feature went to general availability.

The first was about backpressure. When the provider returns HTTP 429 with retry-after: 8, does the user's request sit and wait for eight seconds, or does the caller receive a 503 and retry on its own schedule? Two people answered differently, and both answers were already in the codebase: the vendor SDK retried internally with its own exponential backoff, and the API handler also caught the eventual failure and retried once more. The second question was about conversation history — read from Postgres at the start of every request, or held in the process that owns the websocket. The answer was both, in different code paths, which explained a bug nobody had been able to reproduce, where an edit to an earlier turn sometimes took effect on the next message and sometimes on the one after it. The third was whether callers would be told which vendor answered once the second provider landed. The fourth was whether the per-tenant token budget was enforced before the call using an estimate or after it using real usage.

The word for what that review needed, and did not have, is a reference architecture: an agreed picture of the parts a whole class of system is built from, what each part is responsible for, and how requests and data move between them. It is not a framework you install or a product anyone sells. It is closer to the three-tier diagram — browser, application server, database — that most backend engineers carry around without remembering where they learned it, and it does the same job, which is to let a room full of people say "the session lives here" and have everyone mean the same box. You meet one in a cloud vendor's architecture centre, at the front of a design document, and on a whiteboard during an incident when somebody draws the system from memory to work out which component was supposed to have handled the failure. It exists because a team that has never drawn the picture still has all the parts, arranged by whoever wrote each of them first, which is a different thing from a design.

None of these is difficult in isolation. What made them expensive was that the system had no location where exactly one answer was true. The retry behaviour was distributed across two layers that could not see each other, and the history read was distributed across two layers that disagreed about which of them owned it. A reference architecture is useful precisely because it assigns each of these questions a single home, and the value shows up during incidents rather than during design.

Five decisions, and the layer that exists to make each one#

The layered picture that most production LLM systems converge on is not five tiers because five is a pleasing number. It is five because there are five questions that every request forces, and each answer needs a single owner.

Who the caller is, and whether this request is allowed to proceed, is the gateway's decision. It terminates authentication, resolves the tenant, applies rate limits and admission control, attaches a request identity that everything downstream will carry, and starts the outermost timeout. Nothing below it should ever have to ask who is calling or whether they may.

What the model is asked is the orchestration service's decision. It composes the prompt from a template and the retrieved and stored context, decides the sequence of calls, runs the tool loop if there is one, and owns whatever the user ends up seeing. It is the only layer with a model of the request as an operation rather than as a call.

Which vendor and which model answers is the provider client's decision, and topic 2.18 covered the shape that this takes — a narrow port per capability, resolved at the composition root — the one place in the process where concrete implementations are wired to the interfaces that use them — so vendor identity stays out of the orchestrator. What matters here is the scope of that layer: it owns the call, its serialisation, its transport, and its errors, and it owns nothing that outlives the call.

What survives the request is the state layer's decision, and it is two different stores doing two different jobs. Conversation state, usage receipts and anything with a durable meaning live in a transactional database, usually Postgres. Embeddings and their index live in a vector store, which is a derived artifact that can be rebuilt from source documents and therefore has different durability and backup requirements from the turn history.

Whether the answer was any good is the evaluation and observability plane's decision. It sits across every other layer rather than under one of them, because the record it needs — which prompt version, which model snapshot, which retrieved chunks, which completion — is assembled from facts that four different layers hold and none of them holds alone.

The picture that answers the review's questions#

The diagram worth drawing is not the deployment topology. It is the request path with the data flows marked and the boundary drawn between work that the response depends on and work that it does not.

The solid arrows are the request path: if any of them fails, the user's answer is affected. The heavy arrows are everything computed after the fact, and their failure costs visibility rather than the product.

Read against that picture, the four questions from the review have single answers. The 429 belongs to the provider client only insofar as it is a transport condition on a call that has not yet produced anything; the decision to give up and tell the user belongs to the orchestrator, because the orchestrator is the layer holding the deadline the gateway set. History is read by the orchestrator from Postgres, once, at the start of the operation, and the websocket process holds no authoritative copy. The caller is not told which vendor answered, because vendor identity does not cross the provider client boundary — but the per-call record does capture it, because the evaluation plane needs to be able to ask whether answers got worse when traffic shifted vendors. The token budget is enforced twice for two different purposes: the gateway makes an admission decision from an estimate before the call, and the orchestrator commits the real usage after it, which is a durable fact rather than a control.

The gateway also establishes the context that every other layer reads instead of recomputing.

@dataclass(frozen=True)
class RequestContext:
    request_id: str          # client-supplied if present, else minted here
    tenant_id: str
    principal: str           # user or service account
    deadline: float          # monotonic clock, absolute
    request_class: str       # "chat" | "extract" | "summarise"
    budget_tokens_remaining: int

Six fields, and the discipline is that no layer below the gateway derives any of them from the raw HTTP request. When the orchestrator needs to know how much time is left before the client gives up, it computes ctx.deadline - now() rather than starting a timer of its own, which is how a system avoids having three nested timeouts that nobody has ever reconciled.

The shape settled before the tooling did#

This layering is not a recent invention and it did not come from any vendor's product line. The version most stacks are still variations of was drawn by a16z in 2023 as Emerging Architectures for LLM Applications, and what it replaced is worth remembering, because the failure modes of the thing it replaced explain why the layers are where they are. The first generation of production LLM features were a notebook's worth of code promoted into a web handler: an API key in an environment variable, a prompt as an f-string, and no store other than whatever the surrounding application already had. That shape works, and it stops working at three predictable points — when a second feature needs the same key and nobody can attribute spend, when someone edits the prompt and there is no record of what it said last week, and when the answer is wrong and there is no way to find the request.

Each of those pressures pushed one decision out into its own layer, which is why the reference architecture reads as five boxes rather than as a framework. TensorOps' Emerging Architectures of LLM Applications 2025 walks the same stack with the trade-offs at each boundary spelled out, and the useful thing about reading a five-year-old diagram next to a current one is how little the layers moved while everything inside them changed. Orchestration frameworks came and went, vector databases consolidated, model APIs converged on a common shape, and the set of decisions stayed the same because it is derived from what a request has to do rather than from what tooling exists.

The document ingestion flow is the piece most often left off the drawing, and leaving it off is how index staleness becomes invisible. Chunking, embedding and indexing run on their own schedule, triggered by uploads or by a nightly job, and they write to the same vector store the request path reads. Drawn properly it is a second flow into the same box, with its own owner and its own version stamp, and the reason it matters is that a change to the chunker is a change to the behaviour of every request that runs afterwards, made by a job that no request timing will ever show.

What this picture is not#

It is not a service decomposition. Five layers do not imply five deployables, and treating them that way is the most expensive mistake available here, because it converts every boundary into a network hop with its own serialisation cost, its own failure mode and its own on-call story. A perfectly reasonable production system runs the gateway as a shared platform component and the other four layers as modules inside one Python service, sharing a process and a connection pool. The boundaries are ownership boundaries first — this decision is made here and nowhere else — and only sometimes deployment boundaries. The test for splitting is the ordinary one: different scaling profile, different release cadence, different blast radius, or a different team's on-call rota.

It is not an org chart either, though the two get confused because a diagram with five boxes invites five teams. In practice the gateway is usually somebody else's platform, the evaluation plane is usually shared across features, and the middle three belong to the feature team.

It is also not a data pipeline or a training architecture. Nothing here trains, fine-tunes or serves a model you own; the provider layer is a client to somebody else's inference service. Where a document ingestion pipeline sits — chunking, embedding, indexing — is a genuine question, and the answer is that it is a separate flow that writes to the vector store on its own schedule, sharing a store with the request path but not the path itself. Azure's GenAIOps for organizations with MLOps investments is worth reading if you have an existing MLOps platform and need to see which of its planes map onto this and which do not.

Finally, the evaluation plane is not a dashboard. A dashboard is a rendering of records that already exist; the hard part, and the part that is missing in most systems that look complete, is the record itself.

What the diagram leaves undecided#

The heavy arrows in that diagram carry a claim that has not been justified yet: that trace export, scoring and the cost record can run after the response without changing anything the user or the next request sees. That boundary is the single most consequential line on the drawing, because putting work on the wrong side of it either makes the product depend on infrastructure it should not depend on, or moves a durable fact somewhere it can be lost. The next chapter works out the test for which side a given piece of work belongs on, and it is not the test most teams reach for.