Every span green and the user still saw a failure

A partner team runs the agent; you run the platform it sits on. Their escalation arrives as a screenshot of a conversation ending in "I could not complete that", and one exported trace. You have no access to their code, their index, or their logs, and the person who could give you access is on leave until Thursday. The trace uses their span names rather than yours:

agent.run                     4.10s   OK
├─ llm.call                   1.10s   OK   stop_reason=tool_use
├─ tool.search                0.30s   OK   result_count=0
├─ llm.call                   0.90s   OK   stop_reason=tool_use
├─ tool.search                0.30s   OK   result_count=0
└─ llm.call                   1.00s   OK   stop_reason=end_turn

Six spans, no errors, no retries, no timeouts, 4.1 seconds end to end, and a user who was told the system could not help them. Nothing in this trace is broken in the sense that chapter 4 was about — there is no status to read, no finish reason that flags a truncation, no duration that stands out. The failure is entirely semantic: every operation completed successfully and the sequence of successful operations added up to a bad outcome.

The shape alone gets you most of the way#

Read the trace as a story and it is legible without a single attribute beyond the ones shown. The first model call decided a search was needed and asked for it. The search completed in 0.30 seconds and returned nothing. The second model call saw an empty result set, decided to search again, and asked for a second search. That one also completed in 0.30 seconds and also returned nothing. The third model call had no new information, stopped on end_turn, and produced the sentence the user saw.

That fixes the failing step: this is retrieval. It is not a model problem in the sense of the model reasoning badly, and it is not a latency or availability problem. Whatever the agent should have said, it was working from an empty context both times it tried, and conceding was a reasonable thing to do with nothing in hand.

Two details sharpen it. The identical 0.30-second durations are the retriever's normal p50 — a healthy round trip that examined an index and came back. A retriever that is down usually fails at a different speed: fast, with a connection error, or slow, against a timeout. Two clean responses at the usual latency say the backend answered; they say nothing about what it answered about. And the agent stopping after two attempts rather than looping to its step cap is well-behaved: it recognised that a second empty result was not going to become a third useful one.

Which attribute is missing#

Everything above is about the retrieval step and none of it tells you why the result set was empty. Five causes produce exactly the trace shown, and they belong to four different teams:

The single most valuable one is the query, and specifically whether the two queries were the same. If the agent reissued an identical query after an empty result, the interesting behaviour is in the agent loop — it did not reformulate, and no amount of retrieval work will change that. If it reformulated meaningfully and still got nothing twice, that is strong evidence that nothing in the index would have matched under any phrasing, which moves suspicion to the filters or the corpus. One attribute, present twice, splits the space of causes roughly in half.

The threshold row is the one that most often surprises people, because it is the case where result_count: 0 is actively misleading. A search that examined 812 candidates and found a best match at 0.31 against a min_score of 0.35 is a tuning problem with a one-line fix, and it is indistinguishable — from a result count alone — from a search that examined nothing because the index was empty. Recording what the retriever considered alongside what it returned costs two integers and a float.

The ticket you would not sign#

The instinct after reading the trace is to file against whoever owns the index, because the failing step is retrieval and retrieval is their component. The trace does not support that, and it is worth being precise about why: of the five causes above, two are in the agent's code (the query and the filters), one is in the retrieval configuration (the threshold), and only two are in the index itself. On a multi-tenant retriever, filter and scoping bugs are the most common source of unexpectedly empty results, ahead of anything to do with the documents — the corpus is fine and the query is being asked of the wrong slice of it.

A ticket filed on this evidence buys a week of someone else's time and comes back as "cannot reproduce, index has 4,180 documents for this tenant", which is both true and irrelevant if the query was scoped to a locale with none of them. The defensible version of the same escalation states what the trace establishes — the failure is in the retrieval step, both attempts returned zero, the backend responded normally — lists the causes it cannot distinguish, and asks for the one attribute that would.

Raising on empty results converts a fact into an error#

The proposal that follows from "every span is green and the user still failed" is to make the tool wrapper raise when the result set is empty. Then the span goes red, the error rate moves, the existing alerting catches it, and the trace looks like a failure because it was one.

It is the wrong shape of fix in three ways at once. An empty result set is a legitimate outcome that is frequently correct: on this workload 4.2% of searches return nothing, and 71% of the runs containing one still end with an answer the user accepted, because the agent falls back to a general explanation or asks a clarifying question. Raising turns roughly 1,750 correct behaviours a day into recorded errors, which does not make the failure alertable so much as make the error rate meaningless — and, given the tail-sampling policies from the previous chapter, guarantees that every one of those traces is retained.

It also destroys the distinction the on-call person most needs. After the change, a retriever that is down and a retriever that correctly found nothing both produce an exception on the same span, and the only way to tell them apart is to read the exception message — a string, unindexed, which is where diagnostic information goes to be lost.

The third cost is the one that is easy to miss because it is not about you. The agent currently receives a tool result saying no documents matched, which is information it can act on: broaden the query, drop a filter, ask the user which product they mean, or say plainly that it has nothing. Replace that with an exception surfaced as an error string and the model is now reasoning about a transport failure rather than about an empty search, and its sensible options collapse to retrying or apologising. Instrumentation that changes what the model sees has stopped being observation.

Recording the outcome instead of forcing it into a status#

Span status answers "did this operation complete?" and should keep answering only that. Everything else belongs in attributes, and the alerting belongs on metrics derived from those attributes rather than on status.

def search_docs(query: str, filters: dict, ctx: dict) -> list[dict]:
    with tracer.start_as_current_span("execute_tool search_docs") as span:
        span.set_attribute("search.query", mask(query))
        span.set_attribute("search.query_hash", stable_hash(normalise(query)))
        span.set_attribute("search.filters", json.dumps(filters, sort_keys=True))
        span.set_attribute("search.index_version", INDEX_VERSION)
        span.set_attribute("search.min_score", MIN_SCORE)

        result = retriever.query(query, filters=filters, limit=8)

        span.set_attribute("search.candidates_examined", result.examined)
        span.set_attribute("search.corpus_size", result.corpus_size)
        span.set_attribute("search.top_score", result.top_score or 0.0)
        span.set_attribute("search.result_count", len(result.hits))
        span.set_attribute("search.backend_latency_ms", result.backend_ms)
        # deliberately not an error: an empty result set is an outcome, not a failure
        return [hit.document for hit in result.hits]

Two of those attributes deserve their names explained. search.query_hash is a stable hash of the normalised query, which is what makes "did the agent reissue the same query" a query you can run without reading any user text, and which is safe to use as a grouping key in places where the query itself is not. search.corpus_size is the tenant's document count at query time, so an empty index announces itself in the same span rather than requiring a second investigation on Thursday.

The remaining gap is the user-visible outcome, and it is a product change rather than an instrumentation one. The run above ended with end_turn and a sentence that means failure, and no attribute on any span distinguishes that from end_turn and a sentence that answered the question. Making it distinguishable means having the agent emit a structured resolution — a small enum in the response schema, answered / insufficient_information / needs_human — and copying it onto the root span as agent.outcome. That costs an output field on every response, a schema change other consumers have to tolerate, and a modest number of tokens; it also produces a self-report, which is a signal rather than ground truth, since a model that says answered can still be wrong. It earns its place because it turns "how often do users see a concession, and for which tenants" into a query over spans instead of a sampling exercise over transcripts.

With agent.outcome and the search attributes in place, the alert that catches this class of failure is a derived metric with no reference to span status at all: the rate of insufficient_information outcomes per tenant, and the rate of zero-result searches per tenant per index version, both watched for change points — a shift in the rate against its own recent history, rather than a crossing of some fixed threshold, since the level that is normal differs by tenant. Neither one fires on a single legitimate empty result, and both fire the morning a reindex quietly drops a tenant's corpus to zero.

Which leaves one problem, and it is the one that has been implied since the first attribute in this chapter. Fixing the next occurrence requires search.query in the trace — the customer's own words, written into a store read by everyone who might be on call, alongside the account numbers and order ids that the other tools return. The mask() call in the listing above is standing in for the rest of that argument.