A span that records a call is not a span that explains it

The weekly quality sample for the support agent came back at 3.6 out of 5 on Thursday 12 March, against 4.3 the week before. Twenty-two conversations had been rated by two people against a five-point rubric; seven were marked poor, and six of those seven happened between the 9th and the 11th. Nothing had shipped in that window — no prompt change, no model change, no retrieval reindex — so the first job was to hold a poor conversation against a good one from the week before and find the difference. The team had tracing on every model call. This is what a span looked like:

{
  "name": "llm.call",
  "trace_id": "9f2c41d7e8b04a13",
  "span_id": "c4",
  "parent_span_id": "c1",
  "start_time": "2026-03-10T14:22:09.118Z",
  "end_time": "2026-03-10T14:22:11.402Z",
  "status": "OK",
  "attributes": {
    "model": "claude-opus-5",
    "duration_ms": 2284,
    "tokens": 20636
  }
}

Some of the vocabulary in that account is worth pinning down before the rest of the chapter rests on it. A span is the record printed above: one timed operation, carrying a name, a start and an end, a status, and a bag of key-value attributes. Each span records the id of the span it started inside, so the spans emitted while handling one request link up into a tree — a trace — and reading that tree says what the request did, in what order, and where the time went. LLM tracing, the subject of this topic, is that machinery pointed at an application built on a large language model: a span for every model call, carrying the prompt that was sent and the completion that came back alongside the timings, nested inside spans for the agent steps, tool calls and retrieval hops around it.

It exists because a wrong answer from such a system leaves nothing else to read. There is no stack trace for a reply that is fluent and incorrect, no exception to catch when an agent concedes, and no reliable way to run the thing again and watch, because a rerun draws a different sample from a context you no longer have. The trace is the record of what the system was holding at the moment it decided, and it is a record you have to choose to write, since none of it is kept by default. In practice you meet it as a flame graph in a trace viewer — each span drawn as a horizontal bar under its parent, its width proportional to its duration — either from OpenTelemetry, the vendor-neutral instrumentation standard most of this tooling speaks, exporting into Grafana Tempo, Datadog or Honeycomb, or from a purpose-built tool such as Langfuse or LangSmith that draws the same tree with prompts and completions readable in a side panel. Teams tend to acquire one the week after the first escalation nobody can reproduce.

Two engineers spent a morning on it and got as far as: both conversations used claude-opus-5, both took a bit over two seconds, both returned OK. Every span in the poor run looked like every span in the good run, because the fields recorded are the fields that are the same whenever the system is working at all. A span shaped like this answers "did a call happen, and did it fail?" — which is the question you have already answered by the time you are reading traces during a quality regression.

What was actually different, found four days later by adding fields and waiting for the next occurrence, was that the retrieval step had begun returning a different document ordering after a library upgrade, so the rendered prompt carried a different set of five knowledge-base articles. Nothing in the span above could have shown that, and no amount of dashboarding over those three attributes would have got there.

The model you asked for is not always the model that answered#

The model attribute above records what the request asked for. That string is an alias, and an alias is a pointer: claude-opus-5 names a family, and what sits behind it can be re-pointed by Anthropic, by your gateway, or by your own routing code without your request body changing a character. Recording the request side alone means recording your intention rather than the event.

The response tells you what actually ran, and there are ordinary production paths where the two differ. Server-side fallbacks are the clearest: with fallbacks: "default" enabled, a request that the safety classifiers decline is re-run on another model inside the same call, so a request naming claude-opus-5 can come back answered by claude-opus-4-8, with the switch recorded as a fallback content block and a fallback_message entry in usage.iterations. Sticky routing then keeps subsequent turns of that conversation on the fallback model for about an hour. A quality regression concentrated in one tenant, in one three-day window, is exactly the shape a fallback episode makes — and it is invisible if your span records gen_ai.request.model and stops.

So record both. gen_ai.request.model is what you asked for, gen_ai.response.model is what answered, and a query for spans where the two differ is a five-second check that costs nothing and occasionally ends an investigation on the spot.

Two parameters, and they are not the ones you remember#

The instinct from earlier generations is to record temperature and top_p, on the theory that sampling settings explain output variation. On the current Claude models those parameters do not exist — temperature, top_p and top_k are rejected with a 400 on Opus 5, Sonnet 5 and the 4.7/4.8 family. The knobs that remain, and that genuinely change what comes back, are the effort level and the thinking configuration.

output_config.effort runs low through max and controls how much the model thinks and how much work it does before answering; thinking.type is adaptive or disabled, with adaptive being the default on Opus 5 when the field is omitted. Both belong on the span, and so does max_tokens, because on these models max_tokens caps thinking and response text together — a route that quietly moved from effort: "high" to effort: "xhigh" under a config change will produce longer, differently-shaped answers within the same token ceiling, and a span that records neither the effort nor the ceiling cannot show you that.

The general rule this instance of is: record the resolved request parameters, from the request object you actually sent, rather than from the defaults you believe are in force. Configuration reaches the call site through layers — a per-tenant override, an environment variable, a feature flag — and the value in the code you are reading is a claim about what should have happened.

The rendered prompt is not the prompt in your repository#

Capturing "the prompt" usually means capturing the template, which is the one version of it that is already in version control and therefore the one you did not need. What varies between a good run and a bad one is everything the template interpolates: the retrieved documents, the conversation history, the tool results fed back in, the account record the previous step looked up. The rendered prompt — the exact system and messages payload as serialised for the API — is the artefact that makes two calls comparable.

The completion side is the same argument in reverse, with one field carrying disproportionate weight. stop_reason tells you whether the model finished (end_turn), asked for a tool (tool_use), ran out of room (max_tokens), or was declined by a classifier (refusal), and each of those turns into a different investigation. A truncated answer with stop_reason: "max_tokens" is a budget problem; the same visible symptom with end_turn is a model behaviour problem, and without the attribute you cannot tell them apart from the output text alone.

from opentelemetry import trace

tracer = trace.get_tracer("assist.support_agent")


def call_model(client, request: dict, ctx: dict) -> object:
    with tracer.start_as_current_span("chat claude-opus-5") as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.request.model", request["model"])
        span.set_attribute("gen_ai.request.max_tokens", request["max_tokens"])
        span.set_attribute("gen_ai.request.effort",
                           request["output_config"]["effort"])
        span.set_attribute("gen_ai.request.thinking", request["thinking"]["type"])
        span.set_attribute("gen_ai.prompt", serialise(request["system"],
                                                      request["messages"]))
        span.set_attribute("prompt.template_id", ctx["template_id"])
        span.set_attribute("prompt.template_version", ctx["template_version"])
        span.set_attribute("tenant.id", ctx["tenant_id"])
        span.set_attribute("session.id", ctx["session_id"])
        span.set_attribute("agent.step", ctx["step"])

        response = client.messages.create(**request)

        span.set_attribute("gen_ai.response.model", response.model)
        span.set_attribute("gen_ai.response.id", response.id)
        span.set_attribute("gen_ai.response.finish_reason", response.stop_reason)
        span.set_attribute("gen_ai.completion", serialise_content(response.content))
        span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)
        span.set_attribute("gen_ai.usage.cache_read_input_tokens",
                           response.usage.cache_read_input_tokens)
        span.set_attribute("gen_ai.usage.cache_creation_input_tokens",
                           response.usage.cache_creation_input_tokens)
        return response

The two prompt.template_* attributes are worth the four lines. Full prompt capture is expensive at volume, and the template id plus version gives you a cheap grouping key: "show me the calls that used template answer.v7 between the 9th and the 11th" is a query you can run over a redacted tier where the rendered text has been removed, which turns out to be most of the tiers you will actually have.

Four token numbers, and why one is a lie#

The span at the top of this chapter records tokens: 20636. That number is the sum of four different quantities that are priced between five and fifty times apart from each other, and once summed it cannot be taken apart again.

For the call in question the real breakdown was 2,140 uncached input tokens, 18,400 tokens read from the prompt cache — the provider's stored copy of a prefix you have sent before, billed at a fraction of the ordinary input rate — no cache writes, and 96 output tokens. At Opus 5 list rates — $5 per million input, $25 per million output, cache reads at roughly a tenth of input, cache writes at 1.25× — that call costs about 2.2 cents:

Priced from the single summed figure at the input rate, the same call comes out at $0.105 — nearly five times too high, and wrong in a direction that makes a well-cached workload look like a runaway one. The reverse error is just as available: a route whose cache hit rate quietly collapses moves 18,400 tokens from the $0.50 column to the $5.00 column, quintupling real spend while the summed token count does not move at all. Keeping the four numbers separate is what makes both of those visible, and cache_read_input_tokens sitting at zero across repeated requests with an identical prefix is the standard signature of a cache invalidator — a timestamp in the system prompt, a per-request id interpolated too early, a tool list that varies by user.

The ids that make two calls comparable#

Everything above describes one call in isolation. Comparison needs a second call that is legitimately comparable, and finding it means carrying enough context on the span to constrain the search. Five identifiers do most of that work.

The tenant id is the one most often left out and most often wanted. Quality regressions are rarely uniform: a retrieval change that reorders documents hurts customers with large knowledge bases and does nothing to customers with twelve articles, and an aggregate score of 3.6 is the average of one population at 4.4 and another at 2.1. Without a tenant dimension on the span you can see the average move and cannot see which half moved.

The response id earns its place for a different reason. When something goes far enough wrong that you open a support case with the provider, the request identifier is what makes the conversation possible, and reconstructing it after the fact from timestamps is unpleasant.

What a complete span still will not tell you#

None of these fields say whether the answer was any good. A span carrying the resolved model, the effort level, the rendered prompt, the completion, four token counts and five ids describes a call that ran exactly as designed and produced an answer that was confidently, fluently wrong. Quality lives in evaluations and in human rating, and the trace's role is to explain a quality signal that came from somewhere else — which is why the investigation at the top of this chapter began with a rubric score, not with a dashboard.

Nor is the rendered prompt free. A single gen_ai.prompt attribute on this workload averages 9 KB once retrieved documents are inlined, and at 40,000 requests a day with roughly five model calls each, that is about 1.8 GB of prompt text a day before anything else on the span. It also lands the customer's own words in a system read by more people than your production database is, which is a decision that gets made once, deliberately, and is the subject of the last chapter of this topic.

The narrower limit is that the span above is still a span about one call. It tells you what was asked and what came back, and it says nothing about how many other calls happened in the same request, which of them was a retry of which, or which tool result produced the argument that this call got wrong. Those are questions about the relationships between calls, and a workload with exactly one model call per request never has to ask them. The next chapter is about the point where that stops being true.