One span per request works until the second model call

The classification service had been instrumented with one span per HTTP request for eight months and nobody had ever wanted more. POST /v1/classify takes a support ticket, sends one message to claude-haiku-4-5 with a fixed system prompt and an enum-constrained output schema, and returns one of eleven categories. The span carries duration_ms: 640 at the median, gen_ai.usage.input_tokens: 1900, gen_ai.usage.output_tokens: 12, gen_ai.response.model: claude-haiku-4-5. Every one of those numbers is exactly true of exactly one model call, because there is exactly one model call. The span and the call are the same object seen from two angles.

That is the case where the flat instrumentation is not a compromise. It is worth naming precisely, because the argument for span trees is usually made as though request-level spans were always wrong, and teams who can see that their own service is the simple case reasonably conclude that the whole argument is overblown. The condition is narrow: one model call, no retries, no tools, no second hop. Under it, a request-level span is a call-level span with a different name, and adding a child span underneath would produce a tree of depth one with duplicated attributes.

The same team shipped POST /v1/answer in the same codebase, with the same middleware, and therefore with the same span. That inheritance is worth noticing, because nobody decided it. The request-level span comes from the web framework's instrumentation, which was installed once and correctly produces one span per inbound request; the model attributes were added onto it later by someone who had a token count in hand and an obvious place to put it. The shape of an LLM service's telemetry ends up being a decision about HTTP servers made before the service existed. Here is a real one:

{
  "name": "POST /v1/answer",
  "trace_id": "b71e0c4a99f2d385",
  "span_id": "a1",
  "parent_span_id": null,
  "status": "OK",
  "attributes": {
    "http.route": "/v1/answer",
    "http.status_code": 200,
    "tenant.id": "acme",
    "duration_ms": 11412,
    "gen_ai.request.model": "claude-opus-5",
    "gen_ai.usage.input_tokens": 41200,
    "gen_ai.usage.output_tokens": 1830
  }
}

Nothing here is false. Every attribute is an accurate total for the request. The problem is that each of them is now a sum over an unknown number of terms, and the middleware that produced it has no way to tell you how many terms there were.

An addition you cannot invert#

41,200 input tokens for one request is either alarming or unremarkable, and which one depends entirely on the decomposition. It could be a single call with a 41,200-token prompt, which would mean something is stuffing the entire knowledge base into context and the fix is a retrieval budget. It could be four calls at roughly 10,000 each, which is a normal tool-using trajectory. It could be two calls at 6,000 and a third at 29,000, which would mean a tool returned an enormous result that got fed back verbatim.

For this particular request it was four calls, at 6,100, 9,400, 12,300 and 13,400 input tokens. That progression is the signature of a conversation that carries its whole history forward on every turn, which is normal and which also means the marginal cost of step four is more than double the cost of step one. You cannot see the progression, or reason about whether the prompt cache is absorbing it, from the number 41,200.

The latency has the same property. 11.4 seconds is compatible with one slow generation, with four calls averaging 2.4 seconds plus 1.8 seconds of tool time, or with three fast calls and one call that was retried twice. Those three stories point at three unrelated fixes — a smaller output budget, nothing at all, and a timeout misconfiguration respectively — and the request span is equally consistent with all of them.

The bottom five rows are not exotic questions. They are the first five things anyone asks when a request behaves oddly, and each of them is a question about a part of the request rather than about the request. That an attribute on the whole cannot answer a question about a part is arithmetic rather than a limitation of any particular tracing backend, and no richer set of request-level attributes fixes it — gen_ai.usage.input_tokens_max and gen_ai.usage.input_tokens_count would let you reconstruct a little, and at that point you are storing a badly compressed histogram instead of the spans that would have given you the data.

What the trajectory looks like when the parts are separate#

The structure that chapter 1 of Agent observability established for agent runs applies directly here; what changes for an LLM service is which boundaries deserve a span. Rendered as a tree, the request above reads:

POST /v1/answer                                    a1   11.41s   200
└─ agent.run  support_agent                        b1   11.38s   steps 3
   ├─ agent.step 1                                 c1    2.14s
   │  └─ chat claude-opus-5                        c2    2.11s   in 6,100   out 210   tool_use
   ├─ agent.step 2                                 d1    4.02s
   │  ├─ chat claude-opus-5                        d2    2.36s   in 9,400   out 180   tool_use
   │  ├─ execute_tool lookup_order                 d3    0.31s   ok
   │  └─ execute_tool search_docs                  d4    1.29s   ok   result_count 5
   │     └─ db.query  vector_search                d5    1.21s   ok
   └─ agent.step 3                                 e1    5.22s
      ├─ chat claude-opus-5                        e2    1.44s   in 12,300  out 96    tool_use
      ├─ execute_tool lookup_account               e3    0.42s   ok
      └─ chat claude-opus-5                        e4    3.31s   in 13,400  out 1,344 end_turn

Reading down the in column gives the prompt growth. Reading the durations gives 9.22 seconds of model time against 2.02 seconds of tool time, so the answer to "should we optimise the vector search" is no. The out column shows that three of the four calls produced almost nothing — 210, 180 and 96 tokens, all of them tool-call arguments — and the last one produced 1,344, which is where the output cost lives. None of that is available from a total.

Two naming details are worth adopting rather than inventing. The OpenTelemetry semantic conventions for generative AI — the published vocabulary of span names and attribute keys that instrumentation libraries agree to emit, so that telemetry from different frameworks is queryable together — name model-call spans chat {model} and tool spans execute_tool {tool_name}, which means a query for "every model call in the last hour regardless of which framework emitted it" is a prefix match rather than an ever-growing list of your team's naming decisions. And the root stays: the HTTP span is not replaced by the tree, it is the thing the tree hangs from, and it remains the join key to your access logs, your load balancer metrics and your customer-facing request id.

Attempts are siblings, and that is most of what the tree is for#

The trajectory above has one call per step, which makes the tree tidy and hides the case that motivates it most often. When a call fails and is retried, the retry is a second chat span under the same step, and that placement carries a fact no attribute on the request could:

   └─ agent.step 3                                 e1    9.87s
      ├─ chat claude-opus-5                        e2    3.00s   ERROR  attempt 1  APITimeoutError
      ├─ chat claude-opus-5                        e3    3.00s   ERROR  attempt 2  APITimeoutError
      └─ chat claude-opus-5                        e4    3.31s   ok     attempt 3  in 13,400  out 1,344

Three spans, one step, one logical model call. The llm.attempt attribute makes the count queryable without walking the tree, and the sibling structure makes it unambiguous: three chat spans under one step are three tries at one thing, whereas three chat spans under three steps are three deliberate turns of the loop. That distinction changes what you do next. Three attempts point at a timeout or a rate limit; three steps point at an agent that is taking too many turns to converge. Rolled up to the request span, both appear as a slightly larger token total and a slightly longer duration.

Retries are also where the totals start being wrong rather than merely undecomposable. A failed call returns no usage, so its tokens are absent from gen_ai.usage.input_tokens while its seconds are firmly present in duration_ms — which means the request span shows a request that spent nine seconds and eleven thousand tokens, and the implied tokens-per-second is a number about nothing. Cost per second, tokens per second, and any efficiency ratio computed at the request level are all quietly contaminated by every attempt that did not return a body.

Which boundaries earn a span#

A tree can be drawn at any granularity, and the temptation once you have decided to nest is to nest everything. The boundaries that pay for themselves are the ones where a question changes hands: the run (owned by your service), the step (owned by the agent loop), the model call (owned by the provider), the tool call (owned by whatever the tool talks to), and any network call inside a tool that can independently fail or be retried.

def run_agent(user_input: str, ctx: dict) -> str:
    with tracer.start_as_current_span("agent.run support_agent") as run_span:
        run_span.set_attribute("tenant.id", ctx["tenant_id"])
        messages = [{"role": "user", "content": user_input}]

        for step in range(1, MAX_STEPS + 1):
            with tracer.start_as_current_span(f"agent.step {step}"):
                response = call_model(client, build_request(messages), ctx | {"step": step})
                messages.append({"role": "assistant", "content": response.content})

                if response.stop_reason != "tool_use":
                    run_span.set_attribute("agent.steps", step)
                    run_span.set_attribute("agent.stop_reason", response.stop_reason)
                    return text_of(response)

                results = []
                for call in tool_calls(response):
                    with tracer.start_as_current_span(f"execute_tool {call.name}") as t:
                        t.set_attribute("tool.call_id", call.id)
                        results.append(dispatch(call, t))
                messages.append({"role": "user", "content": results})

        run_span.set_attribute("agent.stop_reason", "step_cap")
        return FALLBACK_TEXT

The step span is the one people leave out, because it wraps no I/O of its own and its duration is only the sum of its children. It earns its place anyway, because it is the only thing that groups a model call with the tool calls that model call requested. Without it, chat and execute_tool spans are flat siblings under the run, ordered by timestamp, and "which tool calls came out of which model call" becomes an inference from start times rather than a fact you can query — which is a small version of exactly the loss this chapter is about.

Depth beyond that rarely helps. A span per message in the array, or a span per token, produces trees nobody reads and volume everybody pays for.

One arithmetic habit has to be unlearned along with the flat span, and it catches people the first time they read a tree with parallel work in it. Child durations do not sum to the parent's duration once anything runs concurrently. In the listing above, step 2 lasts 4.02 seconds while its children total 3.96 seconds, and the small gap is the loop's own overhead — but the moment lookup_order and search_docs run at the same time, the children can total five seconds inside a step that lasted two. "Where did the time go" therefore becomes a question about the critical path rather than about a sum, which is why trace viewers draw the flame graph the way they do and why a table of total time per span name is a misleading summary of a concurrent trajectory.

The tree is not free and it is not causality#

This trajectory is 12 spans for one request. Runs with more steps and parallel tool use land between 20 and 30, and at 40,000 requests a day that is roughly one million spans a day. Most vendors price ingest per span rather than per byte, so moving from one span per request to a full trajectory is a 25× change to that line of the bill, not a rounding error — which is what makes the sampling policy in chapter 4 a design decision rather than a default you inherit.

The other thing to be clear about is what a parent-child edge means. It means your code opened the child span while the parent was the active span. It does not mean the parent caused the child in any semantic sense, and it does not mean the child is a sub-part of the parent's work in any sense the model would recognise. When execute_tool search_docs sits under agent.step 2, that is an assertion your instrumentation made because you wrapped the dispatch loop inside the step; no network call carries that relationship, and no framework can infer it for you. The structure is as true as the code that draws it.

Which is the assumption the next chapter breaks. Every parent-child edge above depends on "the parent was the active span when the child started", and active is a property of an ambient context that lives on the executing thread or task. The moment tool dispatch stops running on the same task that opened the step span, that ambient context is no longer what you think it is, and the tree comes apart in a way that leaves every duration in this listing perfectly correct.