The depth you can attribute is shallower than the depth you can afford

The brief service has a second surface. Alongside the Monday brief on eighteen companies there is an interactive endpoint where an account manager asks a question about the competitive set and gets an answer while they are still on the call. The research for it is pre-indexed, so workers query a store rather than fetching the web, and a run does about four seconds of actual work. Product holds it to a p95 of 20 seconds, which is the number below which the account manager does not switch tabs. A promise of that form is a service level objective, or SLO: a target on some measured property of a service — latency here, availability elsewhere — that the team commits to and is held to afterwards. The p95 is the value that 95 runs in a hundred come in under.

The proposal on the table is a fourth delegation level — sector, sub-sector, segment, worker — to handle a competitive set that has grown past sixty companies. Two quantities decide whether that proposal is buildable, and only one of them is in the design doc. The first is how much time a delegation boundary costs, which has to be answered at the tail rather than at the middle. The second is how deep the run's own records reach, which has nothing to do with time and turns out to bind first.

A boundary's cost at the mean and at the tail#

The argument for the fourth level, verbatim from the design doc: "Each boundary adds about 2.5s. Four boundaries is 10 seconds of coordination, which leaves 10 seconds of the 20-second SLO for the actual work. It fits. We can improve tracing in parallel." Both sentences are wrong, and they are wrong in different ways, which is why the proposal survived three reviews.

Start with where the 2.5 seconds comes from, because the number is real. A boundary's overhead is the parent's inference call to decompose the task, plus serialisation of the delegation brief, plus deserialisation and validation of the return. Two of those three are microseconds of ordinary computing. The inference call is the whole of it, and its duration varies with how much the parent decides to say — which is a length the model chooses at generation time rather than a length the system fixes. Occasionally the parent decides to say a lot.

That is the mechanism behind a long right side, and it produces a distribution whose middle and whose tail are far apart. The mean per-boundary overhead is 2.5 seconds. The p95 per boundary is 6.0 seconds. A budget written against the first of those numbers and checked against the second is the single most common way an agent architecture misses a latency target while every dashboard stays green: at four boundaries the mean coordination cost really is about 10 seconds, and the mean total is around 14, comfortably inside 20. That is the number the doc's author was watching, and it is a true number about a quantity nobody promised.

The naive upper bound on the tail is the one to reach for first, because it is arithmetic anyone can check. Four boundaries at 6.0 seconds each is 24 seconds of pure coordination, which exceeds the entire SLO before a single company has been looked up. If the true p95 were anywhere near that bound, the proposal is dead without further measurement.

Why the naive tail bound is usually loose and is nearly tight here#

Adding four p95s to bound a total is normally far too pessimistic, and knowing why is what makes the exception legible. For a boundary to land at or above its own p95 is by definition an unusual event. For four independent boundaries to land in their slow tails on the same run is much more unusual still, because independent unlikely events have to coincide. So the 95th percentile of a sum of independent hops sits well below the sum of the 95th percentiles: most slow runs are slow at one hop and ordinary at the other three, and one slow hop does not carry the total to the bound.

Two properties of this system break that reasoning, and they break it in the same direction. Both are worth understanding rather than memorising, because each one tells you which other systems the exception applies to.

The first is that the hops are not independent. They run against the same model endpoint within the same few seconds of each other, so the thing that makes one hop slow — the endpoint being slow — is a condition the whole run is sitting inside. Under that condition the hops do not fail to coincide; they coincide by construction, and their tails arrive together.

The second matters more, and more importantly it generalises, because it is a property of fan-out rather than of the provider. A parent that dispatches k children in parallel does not wait for a child. It waits for the slowest child, and the maximum of k draws sits far to the right of a single draw. The reason is the same compounding that runs through this whole topic, used the other way round: if one child comes in under some time with probability p, then all k do with probability p to the power of k, and that number falls as k rises. The time by which every child has arrived, with the confidence a p95 demands, is therefore later than the time by which one child would have. Adding a level multiplies the number of parents doing that waiting, so tail latency grows faster than linearly in depth while the mean grows linearly.

Measured on the regression suite rather than derived, that is what the numbers look like:

Read the second and third columns as rates of change rather than as levels. The mean column steps by the same 2.5 seconds at every level, which is what a per-boundary constant looks like when you add boundaries. The p95 column does not: its first two steps are close to each other and its last step is roughly twice either of them. That accelerating step is the maximum-of-k effect arriving, and it is the difference between a cost you can extrapolate and one you cannot.

The fifth column says the same thing in the units that matter to whoever signs the SLO. Headroom against the 20-second target falls from 10.6 seconds at one level, to 5.9 at two, to 1.4 at three: roughly a halving, and then a collapse. A budget being consumed by a linear cost would give up an equal slice per level, and this one does not. A team reading only the pass/fail column sees three consecutive passes and concludes there is room for a fourth level; a team reading the headroom sees the last usable level arrive and go.

The fourth column is measured rather than added, and it has to be. The run p95 is not the coordination p95 plus the work p95, because percentiles do not compose that way — at depth 1 the coordination p95 is 6.0 seconds against about four seconds of work, and the measured run p95 comes in at 9.4, below what adding them would give. Pretending otherwise is the same class of error as the mean-based budget, made in the opposite direction.

The answer to the design doc is in the fourth row. At four levels the endpoint misses its SLO by 7.2 seconds, which against a 20-second target is a miss of 36%, while the mean-latency dashboard reads 14 seconds and looks healthy. It is missed by design rather than by regression, and that distinction decides what work could fix it. A regression is a gap between what a system does and what it did, so optimising closes it. A design-time miss is a gap between the coordination alone and the entire budget, and no amount of making the work faster helps, because the work is not what is over.

The constraint in the sixth column#

Three levels passes the latency test with 1.4 seconds of headroom. If latency were the only constraint, three levels would be shippable and tight. It is not shippable, for a reason that appears nowhere in the design doc, and the sixth column of the table is where it shows up: correlation IDs propagate exactly one level below the orchestrator.

A correlation ID is the single identifier stamped on every record a run emits, so that records scattered across agents and machines can be gathered afterwards into one story. A span is one timed record of one unit of work, with a start, an end, and a pointer to the span that caused it. Together they are what turns a pile of logs into a tree you can walk.

The tracing library here was wired up when the tree was two levels deep, and it does the obvious thing: the orchestrator mints a trace ID and passes it in the delegation brief, and the direct children attach it to their spans. Nothing below that is instrumented, because at the time there was nothing below that to instrument. The gap is not a bug anybody introduced. It is a piece of plumbing that was correct when written and was never extended when the shape it described changed.

The cost of that gap is worth being precise about, because "debugging is slower" understates it by a category. The per-boundary verdicts of chapter 3 — schema-valid, brief-satisfied, usable-downstream, recorded per hop — stop being computable below the instrumented level, since a verdict with no trace ID cannot be attached to the run it belongs to. That has a consequence upward. The shared-origin detection of chapter 4 works by counting, for each failed run, how many of its boundaries scored not-clean, and reading a spike at zero as the signature of a defect that entered above every boundary. An uninstrumented boundary contributes no verdict at all, so a run whose bad hop was uninstrumented also reports zero not-clean boundaries — the same signature, from a different cause.

That is the step to hold on to. The instrumentation that told the team its failures were correlated is degraded by the very depth it was used to argue about, and it is degraded in the optimistic direction: unattributed boundaries look clean rather than looking unknown. A tree deeper than its tracing does not merely resist debugging. It reports reliability numbers that are wrong, and wrong in the direction that argues for going deeper.

That also disposes of the design doc's second sentence, which proposed to improve tracing in parallel with shipping the fourth level. Tracing work and depth work are not independent tasks that happen to be running at the same time. The instrumentation is what would tell you whether the deeper tree was behaving, and until it lands, the deeper tree's own numbers are biased towards looking fine. Shipping first means the window in which the change is unevaluable is exactly the window in which the decision to keep it gets made. Whichever of the two pieces of work is genuinely urgent, they have to be ordered rather than overlapped.

So the cap is two, and it is decided by attribution rather than by latency. That ordering is the general shape rather than a quirk of this system. Latency and cost budgets scale smoothly and give a few seconds of warning as you approach them. Observability capability is a step function you are either inside or outside of, and it is usually built for the depth you had rather than the depth you want.

A cap written as code, with the conditions that would raise it#

A depth cap that lives in a design doc gets exceeded by the next person who adds a supervisor because a branch got wide. Making it an assertion at dispatch costs six lines and turns the policy into something that fails at build time.

MAX_DEPTH = 2  # decided by trace attribution, not by the latency budget

def dispatch(brief: DelegationBrief, depth: int) -> dict:
    if depth > MAX_DEPTH:
        raise DepthCapExceeded(
            f"{brief.boundary_id} at depth {depth}; cap is {MAX_DEPTH}. "
            "See docs/depth-policy.md for the conditions that would raise it."
        )
    return invoke(brief)

The document reference in the error message is not decoration. A cap with no stated release conditions becomes folklore within two quarters: nobody remembers whether it was a latency number or a tracing number, so nobody knows what would justify changing it, and it gets either treated as sacred or quietly ignored. Both of those are worse than an argued number.

It is worth separating the two kinds of thing that produced MAX_DEPTH = 2. The 20-second SLO is a requirement — it came from product, it is about what an account manager will tolerate, and no engineering decision can move it. The cap is a decision: it was derived from that requirement plus the state of the tracing, and a different tracing state would have derived a different number. Confusing the two is how a cap becomes sacred, since a requirement is not up for renegotiation and a decision always is.

The cap rises to three when all of the following hold. The four conditions below are the ones this cap was derived from, written as things that can be checked rather than as aspirations.

The correlation ID reaches every boundary, verified by a test asserting that a run emits spans carrying the run's trace ID at every depth. A test rather than a dashboard, because a dashboard shows what was recorded and the failure here is that nothing was recorded.

A trace query can name the boundary at which a failed run went wrong in at least 95% of the last 40 failed runs. The metric is attribution coverage rather than trace volume, and the difference matters: a tree can emit a great many spans and still leave the failing hop unnamed.

The denominator of that condition is worth noticing, because it decides how long the condition takes to satisfy. Attribution coverage is computed over failed runs, not over all runs, and failures are a small fraction of traffic. The last 40 failed runs are therefore many weeks of production rather than a day of it, which means the condition cannot be cleared by a deploy on Friday and re-read on Monday. That slowness is a feature. A metric over a rare population is exactly what stops a team from declaring an observability gap closed on the strength of one successfully traced incident.

Any single sub-tree can be replayed in isolation from its recorded DelegationBrief. That requires two things which are easy to have separately and rarely have together — briefs persisted verbatim, and a delegate that can be invoked with a brief and nothing else.

And the measured run p95 at the proposed depth, on the regression suite, sits at or below 17 seconds. Against a 20-second target that leaves 15% of the SLO as burn margin — room for the slow week that arrives without a deploy — rather than shipping at 18.6 and treating the SLO as a target to touch.

That fourth condition is the one that keeps the latency constraint honest across the growth this whole proposal exists to serve. Sixty companies rather than eighteen means wider fan-out at every level, and by the maximum-of-k argument above, wider fan-out means the parent waits on the maximum of more children. The same depth therefore costs more tail latency at the larger set than at the smaller one. A cap validated at eighteen companies is not automatically valid at sixty, and re-measuring is cheaper than re-litigating.

Breadth is cheap where depth is expensive#

The cap is on depth and not on breadth, and conflating the two produces the wrong response to every capacity problem. Sixty companies under two levels means each of the six segment supervisors dispatches ten workers instead of three. That costs fan-out latency by the maximum-of-k argument, and it costs some reconciliation quality, and it adds no depth, no restatement of the premise, and no unattributable boundaries. Breadth is comparatively cheap and depth is comparatively expensive, and the reflex to fix a wide branch by inserting a level is the reflex this cap exists to interrupt.

Two is also not a statement about how deep trees can be in general. It is what this system can operate given the tracing it has and the SLO it is held to. A batch pipeline with a four-hour window and full span propagation might reasonably run four levels, and nothing in the arithmetic above forbids it. The cap is a function of four inputs — error budget, latency SLO, cost limit, and observability reach — and it expires whenever one of them changes. A number carried from one system to another has left all four behind.

Nor is the problem fixed by better dashboards, which is the response the sixth column most often provokes. A dashboard is a view over recorded data, and at depth 3 the data is not recorded. Adding panels over an unattributed span stream produces a more confident version of the same blindness, which is strictly worse than the blindness alone. Retries do not help either — chapter 4 priced those out against a premise that retrying re-sends unchanged — and neither does inserting a validation layer, which would itself be another boundary and would need its own trace ID to be worth anything.

What the cap establishes is a ceiling, and a ceiling is a weak thing to know. It says that a third level is not available on this system today, and it says nothing at all about whether the two levels you are permitted are worth having. A two-level tree that did not need its second level is still paying for two extra inference calls, two summarisation steps and two more places a constraint can go missing, and it will pass every check in this chapter while doing so.

The positive condition — what has to be true of a branch before a supervisor over it earns its keep — is a separate question, and the boundaries are now priced well enough to state it as a threshold rather than as a preference.