Spans keep the parent-child structure that log lines discard

An agent calls a tool, gets an error back, and calls the same tool again. Two records land in whatever the service writes to. Whether they describe one agent step whose network request was retried, or two agent steps each of which decided independently to make the call, is a question about the run rather than about either record — and the answer decides how many model calls were billed, how many of the loop's step budget were spent, and whether the model was ever shown the error at all. A flat stream of log lines cannot answer it, and the reason it cannot is structural rather than a matter of logging discipline.

Agent observability is the practice of instrumenting an agent so that every run leaves behind a structured record of its trajectory — the actual sequence of steps that run took, with each model call, each tool call, each retry and each error kept in its place relative to the others rather than flattened into a list of messages. It exists because an agent decides its own control flow while it is running, so the same deployed code takes three steps on one claim and thirty-one on the next, and reading the source tells you nothing about which of those happened on the claim someone is asking about. In practice you meet the record as a tree in a viewer — Langfuse, LangSmith, Grafana Tempo, Datadog, or whichever tool the company already runs for its services — where a single run is a stack of nested bars that expand to show durations, token counts, arguments and errors hanging off each one. That record is called a trace, and everything later in this topic is a question put to it: what each step cost, where a wrong value entered, whether the run can be made to happen again, so what the record contains and what shape it has are settled before any of those questions can be asked.

Two runs that leave the same log lines#

Here is what a log stream held for one run of a claims-triage agent, run_7f3a, filtered to a single tool:

{"ts":"2026-03-04T09:41:12.338Z","level":"info","msg":"tool call","run":"run_7f3a","tool":"estimate_repair","claim":"CLM-88214"}
{"ts":"2026-03-04T09:41:13.902Z","level":"warn","msg":"tool error","run":"run_7f3a","tool":"estimate_repair","status":503}
{"ts":"2026-03-04T09:41:14.410Z","level":"info","msg":"tool call","run":"run_7f3a","tool":"estimate_repair","claim":"CLM-88214"}
{"ts":"2026-03-04T09:41:16.771Z","level":"info","msg":"tool ok","run":"run_7f3a","tool":"estimate_repair","ms":2361}
{"ts":"2026-03-04T09:41:17.004Z","level":"info","msg":"model call","run":"run_7f3a","in":9411,"out":221}

Two tool call records, one error between them, one success. That is consistent with two entirely different things having happened, and nothing in the file distinguishes them. In the first story, the retry wrapper caught the 503 and re-issued the HTTP request itself: one agent step, two network attempts, and the model never learned that anything went wrong. In the second story, the wrapper was bypassed — perhaps because 503 was not in its retryable set on that code path — the error was formatted as a tool result and handed back to the model, and the model decided on its own to call estimate_repair again with identical arguments: two agent steps, two model calls, and one step consumed against the loop's cap.

Work out why the file cannot settle it, because the reason is more specific than "logs are unstructured". Every field on every one of those lines is a fact about the line itself: a timestamp, a level, a tool name, a status. The thing that separates the two stories is not a property of any line — it is whether the second tool call sits underneath the first one's step or beside it. That is a relation holding between two records, and a record has nowhere to put a relation. Adding fields does not help, because whatever field you add is another fact about one line.

The obvious rescue is to lean on order: the lines are in a file, files have an order, and in the second story a model call would sit between the two attempts. That rescue fails, and the model call record at the end of the excerpt is the demonstration. It is written by a wrapper that batches records and flushes them asynchronously, so the moment a line is appended to the file is decided by the flusher's timer rather than by the moment the event occurred. Position in the file is therefore a fact about the log writer, not about the run, and adjacency in the file is not evidence of adjacency in time. Anything you infer from the ordering of two lines written by two different code paths is an inference about buffering.

The distinction is not academic. The first story costs 2.4 seconds of extra latency and nothing else. The second story costs an extra model call, a step from a budget of ten, and a small but real chance that the model interprets the error text as evidence about the claim rather than about the network. Answering "which one is it?" from the flat stream requires reading the source of the retry wrapper and reasoning about what it would have logged, which is a way of saying the stream does not contain the answer and the source code is being used as a substitute for a record. That substitution is only available to someone who knows which build was deployed when the run happened, which is a different and worse question.

The parent pointer#

A span is a named interval of work with a start time, an end time, a unique id, a pointer to the id of the span that caused it, and a bag of attributes. A trace is the set of spans that share a trace id — usually one agent run. The pointer to the parent is the entire difference. A log line records that something happened; a span records that something happened inside something else, and that containment relation is what carries the information the flat stream threw away.

The pointer is what makes containment storable, and it is worth being exact about the trick. A relation between two things cannot be a field on one of them, but a reference can: each span carries the id of the span it happened inside, so the relation is reconstructed by following pointers rather than encoded on any single record. That is the same move a linked list makes, and it has the same consequence — the structure is only as good as the pointers, and a span whose parent id is missing or wrong is not a slightly degraded record, it is a record that has left the tree.

Rendered as a tree, the same run reads like this:

run.claims_triage                                trace 4c1e…   6.81s
├─ step.1                                        span a1       1.22s
│  └─ llm.completion  claude-opus-5              span a2       1.19s   in 2,140  out 96
├─ step.2                                        span b1       1.44s
│  ├─ llm.completion  claude-opus-5              span b2       0.81s   in 2,620  out 142
│  └─ tool.fetch_claim                           span b3       0.61s   ok
└─ step.3                                        span c1       4.02s
   ├─ llm.completion  claude-opus-5              span c2       0.39s   in 4,450  out 210
   └─ tool.estimate_repair                       span c3       3.60s   attempts 2
      ├─ http.POST /v1/estimates                 span c4       1.56s   503
      └─ http.POST /v1/estimates                 span c5       2.36s   200

The answer is now a matter of reading rather than inference: two HTTP attempts, both children of a single tool.estimate_repair span, which is itself a child of one step that contains exactly one model call. That is the first story. Had it been the second, the tree would have shown two sibling step spans, each with its own llm.completion child and its own single-attempt tool span, and the total model calls in the trace would have been four rather than three. That count is the point. The two stories are indistinguishable in every per-record field and differ by one model call, and the only thing that produces the extra call is a second step — which is to say the billing difference between the two stories is a consequence of the containment and of nothing else.

The two attempt spans also carry the durations the log stream's own timestamps give for the same two attempts, which is the small piece of corroboration that makes the tree trustworthy rather than merely convenient. The attempts are the same events seen by two mechanisms, and the mechanisms agree.

That corroboration generalises into a property worth relying on. The number of attempts is readable from the tree twice over: once from the tool.attempts attribute, which the retry loop sets on the tool span by counting its own iterations, and once from the number of http.POST children hanging off that span, which the tracer produces from its parent pointers without knowing anything about retries. Two instruments, no shared step, one answer. What that licenses is a record that survives either instrument failing — and, more usefully, it makes a disagreement between them a finding in its own right. An attempt count of one under two child spans means someone opened a span the loop did not count; two attempts recorded under one child span means an attempt was made below the layer that opens spans at all, which is exactly the client-library retry that the chapter on cost reconciliation has to go looking for.

The residual on that is worth stating, because it bounds what a trace can ever be honest about. Both instruments live inside your process, so a retry issued underneath the tracer's reach — inside a vendor SDK, inside a connection pool, inside a proxy — is invisible to both of them and the two will agree with each other while being jointly wrong. A trace is only ever a truthful record of the layer it wraps. That is the same defect as an orphaned sub-agent trace, which is also two records that are each internally consistent and have no pointer between them; the orphan loses a relation across a process boundary and the swallowed retry loses one across a library boundary.

Wrapping the loop#

The agent loop built in 10.3 already has the boundaries that spans want: the run, the step, the model call, the tool dispatch. Instrumenting it is a matter of opening a span at each of those boundaries and letting the tracer maintain the parent pointer for you as a context-local value. The OpenTelemetry SDK does this with a context manager, so the nesting in the code is the nesting in the trace.

The mechanism behind that convenience is worth one sentence, because it is what makes tool-internal work land in the right place later. The tracer keeps a reference to the currently open span in context-local storage — the same storage a thread-local or an async task-local uses — and every new span reads that reference to fill in its parent id, then installs itself as current for the duration of the with block. Nothing is passed as an argument, so a function called three layers down opens a span that is correctly parented without knowing it is being traced.

from opentelemetry import trace

tracer = trace.get_tracer("agent.claims_triage")


def run(user_input: str, claim_id: str) -> dict:
    with tracer.start_as_current_span("run.claims_triage") as run_span:
        run_span.set_attribute("agent.name", "claims_triage")
        run_span.set_attribute("claim.id", claim_id)

        messages = [{"role": "user", "content": user_input}]

        for step in range(MAX_STEPS):
            with tracer.start_as_current_span(f"step.{step + 1}"):
                with tracer.start_as_current_span("llm.completion") as llm_span:
                    response = client.messages.create(
                        model=MODEL, max_tokens=2048, system=SYSTEM,
                        tools=TOOLS, messages=messages,
                    )
                    llm_span.set_attribute("gen_ai.request.model", MODEL)
                    llm_span.set_attribute("gen_ai.response.model", response.model)
                    llm_span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens)
                    llm_span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens)
                    llm_span.set_attribute("gen_ai.response.finish_reason", response.stop_reason)

                messages.append({"role": "assistant", "content": response.content})
                calls = [b for b in response.content if b.type == "tool_use"]
                if not calls:
                    run_span.set_attribute("run.stop", "end_turn")
                    return {"stop": "end_turn", "output": text_of(response)}

                results = []
                for call in calls:
                    with tracer.start_as_current_span(f"tool.{call.name}") as tool_span:
                        tool_span.set_attribute("tool.name", call.name)
                        tool_span.set_attribute("tool.call_id", call.id)
                        output = dispatch_with_retry(call, tool_span)
                        results.append({
                            "type": "tool_result", "tool_use_id": call.id, "content": output,
                        })
                messages.append({"role": "user", "content": results})

        run_span.set_attribute("run.stop", "max_steps")
        return {"stop": "max_steps", "output": None}

The retry wrapper is where the chapter's opening question is actually settled, and it settles it by opening a span per attempt rather than by writing a better log line:

def dispatch_with_retry(call, tool_span, max_attempts: int = 3) -> str:
    last_error = None
    for attempt in range(1, max_attempts + 1):
        with tracer.start_as_current_span("http.POST") as http_span:
            http_span.set_attribute("http.attempt", attempt)
            try:
                result = DISPATCH[call.name](**call.input)
                http_span.set_attribute("http.status_code", 200)
                tool_span.set_attribute("tool.attempts", attempt)
                return result
            except UpstreamError as exc:
                http_span.set_attribute("http.status_code", exc.status)
                http_span.set_status(trace.StatusCode.ERROR, str(exc))
                last_error = exc
        time.sleep(0.25 * 2 ** (attempt - 1))
    tool_span.set_attribute("tool.attempts", max_attempts)
    tool_span.set_status(trace.StatusCode.ERROR, str(last_error))
    return json.dumps({"error": "estimate service unavailable", "status": last_error.status})

Notice which span each attribute is set on, because that placement is the whole design. The status code goes on the attempt span, since a status is a property of one HTTP request. The attempt count goes on the tool span, since a count is a property of the tool call as a whole. Putting the status on the tool span would keep only the last one and quietly erase every failure that preceded a success, which is the shape of instrumentation that reports a healthy system by construction.

Attribute names matter more than they look. The gen_ai.* keys above come from the OpenTelemetry semantic conventions for generative AI, which exist so that a token count from your hand-rolled loop and a token count from a vendor SDK land in the same field and can be summed in the same query. Inventing tokens_in because it is shorter costs nothing on the day you write it and costs a migration on the day you adopt a second framework. AI Agent Observability 2026: Tracing & Monitoring Stack is a reasonable survey of where those conventions currently stand if you want the field list rather than the argument for having one.

Borrowed from RPC tracing, and where the analogy strains#

None of this machinery was invented for agents. Distributed tracing came out of the mid-2000s microservice problem, where a single page load fanned out to dozens of services and the flat per-service logs could not be assembled into a story — the same complaint as the one that opened this chapter, arriving twenty years earlier. Google's Dapper paper in 2010 set the shape that survives today: a trace id propagated through every call, a span per unit of work, a parent pointer, and sampling because you cannot afford to keep everything. OpenTelemetry is the standardised descendant, and an agent trace is a plain OpenTelemetry trace with domain-specific attribute names. Adopting it means inheriting exporters, storage backends, sampling policies and query languages that have been in production for a decade, which is a much better position than designing a bespoke format.

Treating an agent run as a request fanning out to services is a genuine analogy and not a definition, and it strains in three places. Each strain follows from one property of agents that RPC services do not have, and knowing the property is what lets you predict the next place it will strain.

The first property is that the fan-out is chosen at runtime by the model rather than written into your code. An RPC service always calls the same three downstreams, so the number of spans a trace contains is a constant you can put in a dashboard. An agent calls two tools on one claim and eleven on the next, so span counts per trace vary by an order of magnitude, and any alert built on "expected spans per trace" fires on ordinary variation and stays quiet when something is wrong. The second property is duration. Dapper-era tracing assumes spans of milliseconds and traces of under a second; an agent run here takes 6.8 seconds at the median and can run for minutes, so a trace is often still open at the moment somebody wants to look at it, and viewers that assume a completed trace show nothing at all until the run ends.

The third property is the one that decides where your code goes. In an RPC system the parent-child relation travels on the wire: service A calls service B over HTTP, and the call itself is the causal link, so a middleware sitting on the connection can record it without understanding anything about the application. In an agent there is no wire carrying the relation. A tool span is a child of a step because the step's model call asked for that tool, and that fact exists only in your loop's control flow. Nothing observes it from outside. The consequence is a design decision rather than a requirement: the instrumentation has to live inside the loop, at the boundaries the loop itself creates, and a team that tries to add tracing as a middleware will get a flat list of HTTP calls with no steps in it.

Sampling is the one inherited practice that needs deliberate rethinking, and the reason is a mismatch between what the default optimises and what an agent run is worth. Head sampling — decide at the root span whether to keep the trace, typically at a few percent — was calibrated for traffic where traces are numerous, individually cheap and statistically interchangeable, so a random few percent estimates the population well and the ones you drop are replaceable by the ones you keep. Agent runs invert both halves of that. The volume is low enough to keep, since 6,000 a day is nothing to store, and the runs that matter are not interchangeable: an errored run, a run that hit the step cap and a run that cost a dollar are each individually the thing somebody will ask about, and a sampler that cannot tell them apart from a routine run drops them at the same rate as everything else. Keep every run that errors, every run that hits the step cap, and every run above a cost threshold, and sample the rest at whatever rate the storage bill tolerates.

Trace ids have to survive the process boundary#

The tree above is a single process, which is the easy case. Agents rarely stay that way for long. A tool that calls your own internal service, a queued job that finishes the run asynchronously, a sub-agent spawned for a second opinion — each of those crosses a boundary where the ambient context is lost unless something carries it. The mechanism is the W3C traceparent header, which encodes the trace id and the current span id in a single string and which every OpenTelemetry instrumentation for HTTP clients injects for you if you let it. For queues, the same value goes on the message as a property and is extracted by the consumer.

The reason a boundary loses the context is the same reason the context worked so smoothly inside one process. The current span lives in context-local storage, and context-local storage is local to a process — a new process starts with an empty one. So the parent id that a nested function got for free is exactly the thing that does not survive a network hop or a queue, and it has to be moved explicitly, as data, in a place the far side knows to look.

The failure mode when this is skipped is not an error. It is a second, orphaned trace with its own root, containing spans that look perfectly healthy and that no query about the original run will ever return. Chapter three shows what that costs when the orphan is a sub-agent doing billed model work, and chapter five shows what it costs when the orphan is the thing that produced the bad value. Both are the same defect: a run whose parts are not connected is a run you can only partly answer questions about, which is the flat log stream again, wearing a different format.

The questions each source can answer#

Read that table as a gap opening rather than as a list of features. The top three rows are there to establish that the two sources are equal where the question is about the run taken as a whole — a status, a total duration, a set of tool names — because each of those is answerable from facts held on individual records, and both sources hold individual records. The gap opens at the fourth row and never closes, and every question below the line has the same form: it asks about one part of the run relative to another. How many attempts sat under this call, whether a model call sat between two attempts, which model call produced these arguments, which spans belong to this run's total. Those are questions about position in a structure, and the source that stores no structure cannot answer any of them.

The last three rows are the ones that make the rest of this topic possible, and they are all consequences of the parent pointer rather than of the format. Structured JSON logs are still flat: adding a parent_id field to every log line and a span_kind and a duration is not an argument against tracing, it is an implementation of tracing with extra steps and no tooling. Once those fields exist, the queries that read them are tree queries, the viewer that renders them is a tree viewer, and the thing you have built is a tracing backend maintained by one team instead of an ecosystem.

Correctness, metrics and the payloads the tree omits#

A trace tells you what happened and in what order and under what. It does not tell you whether any of it was correct. The tree above shows a run that made three steps, called two tools, retried one of them once, and finished cleanly — and it shows exactly the same shape whether the agent approved a valid claim or approved a fraudulent one for eleven thousand pounds. Correctness lives in evaluations and in assertions over outputs, and a team that reads its trace viewer every morning and feels informed is at risk of mistaking a clean shape for a right answer. The mistake is easy to make precisely because the tree is so legible: the structure it renders is complete and unambiguous, and completeness about structure reads as completeness about everything.

Tracing is also not metrics, and reaching for it as if it were is the common early mistake. A trace is a record of one run, retained for days or weeks and sampled if volume is high; a metric is a pre-aggregated counter or histogram retained for a year and never sampled. "What is the p95 latency of estimate_repair this quarter?" is a metrics question, and answering it by scanning traces is both slow and wrong, because the traces you kept are not the traces that happened. That last clause is the whole objection and it is worth spelling out: any sampler, random or otherwise, makes the retained set unrepresentative of the population in some way, and a percentile computed over a biased sample is a percentile of the bias. The relationship is that spans make good metric emitters — you can derive a duration histogram per tool from the same instrumentation — but the derived series, not the spans, is what you alert on, because the series is computed before anything is dropped.

And the tree above is deliberately thin. Every span carries names, ids, durations, statuses and token counts, and not one of them carries what the model was asked or what the tool returned. That is the version most teams reach for second, because the shape tells you where a run went strange and the payloads tell you why, and the moment those payloads land in the tracing backend the claimant's name, address and medical narrative land there with them — in a system that is read by far more people than the claims database is. Whether that is allowed, and what remains debuggable if it is not, is settled before the first payload is written rather than after.

The pointer, then, is the whole of it. A span costs almost exactly what a log line costs to emit and carries one extra field, and that field is a reference to the record it happened inside. Everything the rest of this topic does — attributing a cost to a step, distinguishing a long run from an expensive call, walking a bad value back to the tool that produced it, replaying a trajectory — is a query over the structure that reference builds, and none of it is available from a stream of records that each know only about themselves.

Which means the first decision is not which vendor to send traces to. It is what a span is allowed to carry, because a trace with the payloads in it answers a much wider class of question and lands a copy of your most sensitive data in a system with a much wider readership, and that trade is made once, at the beginning, whether or not anyone notices they are making it.