A green dashboard is compatible with a wrong answer

The platform team runs forty services, and their question in the architecture review was fair: "You have distributed tracing, per-service SLOs, canary deploys, a mesh doing circuit breaking and outlier ejection, and cost allocation by team. Tell us what an LLM feature needs that we do not already give it." The honest first half of the answer is that most of what they built carries over unchanged, and the reference architecture from chapter 1 is largely their existing platform with three boxes added.

The tracing carries over, and the gateway is their gateway. The SLOs on latency and availability are the right SLOs — a service level objective being the target a team commits to publicly, such as "p95 under 6 seconds, 99.9% of requests succeed" — though the p95 target has to be set against generation latencies rather than CRUD latencies. Circuit breaking against a provider endpoint works exactly as it does against any upstream, and outlier ejection, which pulls an endpoint out of the load-balancing pool once it starts failing more often than its peers, is genuinely useful because provider regional endpoints degrade independently. Cost allocation by team already has the shape you need, since token spend attributes per tenant and per feature the same way compute does. Canary deploys work for the service. Roughly four fifths of an LLM system's operational needs are ordinary service operations, and a team that starts by declaring their existing platform inadequate ends up rebuilding tracing badly.

The gap is narrower and harder than "we need AI observability". It is that a specific class of regression produces no error, no latency change and no deploy, so every control on that list is structurally incapable of firing.

The regression that nothing on the list can see#

Nine months into production, a data pipeline change rebuilt the retrieval index with a chunk size of 1,024 tokens instead of 512, to reduce index size. Retrieval still returned five chunks in 38 ms. The provider still returned 200s. The service still met its p95 of 6.2 s. Nothing deployed in the LLM service at all — the change was in a nightly indexing job owned by a different team, and it landed as a config value in a YAML file. Measured afterwards against a 300-question set with known-correct source passages, recall@5 fell from 0.91 to 0.78 — recall@5 being the share of questions whose known-correct passage appears somewhere in the five chunks retrieval returned — concentrated almost entirely in scanned documents where the larger chunks now spanned two unrelated sections. The regression was detected nineteen days later through accumulated support complaints.

Every control the platform team listed was working correctly and none of them could have fired. Circuit breaking measures failures, and there were none. Canary compares error rate and latency between versions, and both were identical — and no version of the serving code changed anyway. SLOs measure availability and latency. Cost allocation would have shown a small decrease in prompt tokens, which reads as an efficiency win.

The distributed trace for one of those bad requests contains everything except the thing that changed.

{
  "name": "chat.generate",
  "duration_ms": 5840,
  "status": "OK",
  "attributes": {
    "tenant.id": "t_2291",
    "http.status_code": 200,
    "gen_ai.system": "anthropic",
    "gen_ai.request.model": "claude-sonnet-4-5",
    "gen_ai.usage.input_tokens": 2104,
    "gen_ai.usage.output_tokens": 480,
    "retrieval.k": 5,
    "retrieval.duration_ms": 38
  }
}

That span is well-instrumented by ordinary standards. It carries no prompt version, no model snapshot — the dated, frozen model identifier that a moving alias resolves to — no index version, no chunk identifiers, and no completion, which means it can tell you that a request happened and nothing at all about whether the answer was right or what would have to change for it to become wrong. Traces are also sampled — head sampling at 5% is normal at forty-service scale, meaning the keep-or-drop decision is taken at the first span and applied to the whole trace — so even the metadata is present for one request in twenty, and the twenty-first is the one support is asking about.

What is actually different, dimension by dimension#

Two rows in that table are the ones that surprise platform engineers with strong operational instincts. The first is that behaviour changes without a deploy, which breaks the assumption every deployment control is built on: canaries, blue-green, gradual rollout and automatic rollback all key on a release event. Prompts are frequently stored in a database so that product managers can edit them, model aliases like gpt-4o or claude-sonnet-4-5 resolve to snapshots that the vendor rotates on their schedule, and the retrieval index is rebuilt by a job. Three behaviour-changing inputs, none of which passes through your release pipeline.

The second is that regressions concentrate in slices. A conventional performance regression tends to move the aggregate, because the code path is shared. A retrieval regression on scanned PDFs moves the aggregate by the slice's traffic share — 8% of traffic dropping from 0.91 to 0.62 moves an overall mean from 0.91 to about 0.89, which is inside the week-to-week noise of any judge-based score. The mean is not merely a weak detector here, it is a detector that gets weaker as your product gets broader.

Temperature zero is not a reproduction strategy#

The nondeterminism row deserves more than a line, because it breaks a habit that every experienced operator has, which is to reproduce the failure first and reason afterwards. Setting temperature: 0 removes the sampling noise and does not make the call deterministic. Provider-side request batching changes the shapes that flow through the kernels, floating-point reductions are not associative, and mixture-of-experts routing — the architecture in which each token is sent through a small subset of the model's many specialised sub-networks — can depend on what else is in the batch, so identical inputs to a pinned snapshot return identical text most of the time and not always. Running one production prompt 100 times at temperature 0 against a pinned model produced 6 distinct completions in a test the team ran afterwards, with 91 of the 100 identical and the rest diverging after a numeric formatting difference around token 40.

The consequence for operations is that a single re-run proves nothing in either direction. A failure that does not reproduce has not been ruled out, and a success on the second attempt is not evidence that the first was a fluke. Anything you want to claim about behaviour has to be claimed over a set — a fixed suite of 200 to 300 inputs, run before and after, compared as a distribution — and the raw material for that suite is the stored completions from real traffic rather than prompts someone invented.

The same logic gives you a cheap detector for vendor-side drift, which is otherwise the hardest of the seven rows to catch. A canary set of 50 fixed prompts run hourly against the pinned snapshot, with each completion hashed and the hash compared to the modal hash — the value that came up most often — from the previous week, costs roughly $0.30 a day and fires when a provider changes something under a snapshot label. It will produce a low rate of false positives from the nondeterminism just described, which is why the comparison is against a modal hash over many runs rather than against a single golden output.

Pinning is the cheapest control on the list#

Two of the seven rows are addressed by configuration rather than by any new system, and they are worth doing before anything is built.

MODELS = {
    "chat":    "claude-sonnet-4-5-20250929",   # snapshot, never the alias
    "extract": "gpt-4o-2024-11-20",
}
PROMPTS = PromptRegistry(
    source="git",            # prompts are code, released with the service
    pin="sha:8f31c0a",       # the running version, stamped onto every record
)
INDEX = {"name": "docs", "version": "2026-03-14T02:11Z", "chunker": "recursive-512"}

Using a snapshot identifier rather than an alias converts a silent vendor-side behaviour change into an explicit action you take, at the cost of having to take that action — snapshots are deprecated on a schedule, typically with several months of notice, so this is a recurring maintenance item rather than a one-off. Keeping prompts in git rather than in a database costs product managers their self-service edit and buys you a release event to canary against; the compromise most teams settle on is prompts in a store with immutable versions and an explicit publish step that emits a deploy-like event.

The index version is the third of the three, and it is the one that most often has no owner at all. Whoever runs the indexing job knows when it ran; the serving team knows what it served; nobody records which index a given answer was drawn from, so a question like "did answers get worse after the March rebuild" cannot be asked without a join that does not exist. Stamping an index version on every record is a single string on a row, and it converts an argument between two teams into a query.

Pinning also changes the release process in a way worth naming. Once prompts carry a version and models carry a snapshot, a prompt edit and a model upgrade both become deploy-shaped events, which means the platform team's existing canary machinery starts working again — you can route 5% of traffic to prompt version B, compare scores on the slices that matter, and roll back by changing a pointer. Everything the platform team built for code applies to prompts as soon as prompts stop being data that changes without an event.

What this argument is not#

It is not a claim that LLM systems need a separate observability stack. Everything described here is a set of attributes on records you are already emitting, and the correct implementation is usually additional span attributes plus a table, not a second vendor. Teams that conclude otherwise buy an LLM observability product and end up with two systems where the traces live in one and the request context lives in the other, which makes incident work harder rather than easier.

It is also not evaluation in the testing sense. Running a fixed suite of prompts in CI and asserting on outputs is a useful thing that catches prompt edits which break obvious cases, and it would not have caught the index rebuild, because the index rebuild happened in production on a Tuesday night and CI does not run against the production index. Offline evaluation answers "did my change break something"; the plane discussed here answers "did something break". The controls overlap in tooling and not in purpose.

And it is not safety filtering. Guardrails that block prompt injection or PII leakage sit on the request path, as chapter 2 established, and they are a different plane with a different latency budget.

What has to be recorded#

The table above prescribes a set of controls without saying what they are built out of, and every row lands on the same underlying object: a durable, per-call record carrying the prompt version, the model snapshot, the retrieval index version and chunk identifiers, and the completion, joined to whatever outcome labels arrive later. The next chapter builds that record against a concrete case, because the difference between a record that turns a support spreadsheet into a metric and one that produces a number nobody can act on is a matter of specific fields and specific slices.