Diagnosing an empty completion

A response arrives, there is nothing useful in it, and an exception is thrown three frames deeper in code that expected text to parse. That symptom has four plausible causes on a reasoning model, and from the application's perspective they look identical. They are not identical in the response payload, where each leaves a different and unambiguous fingerprint.

This chapter is about reading that payload: the four candidates and how one field comparison tells them apart, why the distribution of reasoning lengths shifts under you when a model alias is repointed, why a limit that looks healthy on a dashboard has no headroom, and which two standing alerts catch the next shift before a user does. The diagnostic takes about ninety seconds once you know what to look at, and it is worth running before anyone opens a prompt file, because three of the four branches lead somewhere the prompt has no bearing on.

Four candidates, one field#

Here is a captured response from an agent endpoint configured with max_tokens of 4,096:

{
  "id": "msg_01Qw7t",
  "model": "agent-lg-2026-05",
  "content": [],
  "stop_reason": "max_tokens",
  "usage": {
    "input_tokens": 6214,
    "output_tokens": 4096,
    "reasoning_tokens": 4096
  }
}

Reading it turns on one fact about how it was produced. Deliberation and answer are not two calls but one stream of tokens with a single counter running over it, and the deliberation comes first, because the whole purpose of it is to be available to the answer that follows. A limit on that counter stops whichever part of the stream happens to be in progress when the count is reached. If the stream is cut before it makes the transition from deliberating to answering, the record of what was spent is complete and correct, while the array that was going to hold the answer has never had anything appended to it.

Everything needed to make the diagnosis is in those seven lines, and it is three separate facts rather than one. The finish reason is length-based, which rules out the model having stopped of its own accord and rules out a stop sequence having matched: a limit ended this generation. output_tokens is exactly the configured limit, which rules out some other limit having been responsible — a context window, a proxy timeout, a per-minute cap — because a generation halted by any of those would have stopped at whatever count it had reached rather than landing on 4,096 precisely. And reasoning_tokens equals output_tokens, which rules out any of the stream having gone into the answer, since every one of those 4,096 tokens is accounted for on the deliberation side. The content array is empty because there was nothing left to write it with.

That reading holds for this response and for a provider that meters deliberation and answer against a single output limit. Where reasoning is drawn from an allowance of its own, the same equality says something different — that deliberation used up its own budget — and the comparison worth making is against that separate limit rather than against max_tokens. The field names are the same in both cases, so the thing to establish before treating the equality as a diagnosis is which of the two you are billed and metered under.

Set that against the alternatives, each of which leaves a different fingerprint:

Each of those fingerprints follows from where in the request the failure happened, which is what makes the table something to derive rather than memorise. A prompt too long for the window fails before generation starts: the provider counts the input, finds it does not fit alongside the output that was asked for, and rejects the request. Nothing was generated, so there is no completion-side accounting to report, and that is why the usage payload is either absent altogether or carries an input-side count and nothing else. The absence is not an omission by the provider; there is genuinely nothing to put there.

A stop sequence is the opposite situation. The model had begun the answer and was emitting it token by token when one of the configured strings matched, so there is visible content, it ends where the match occurred rather than where the thought did, and the counter is well short of its limit because nothing was forcing the stream to run that far. Content present and output comfortably under the allowance is a combination the other three causes cannot produce, which is why one glance at those two fields settles it.

The last row is the one worth being able to rule out quickly, because it produces the most confused incident channels. Nothing on the model's side went wrong in that case, so the provider's own request log shows a response that completed normally with a full token count, and an investigation that starts from the provider's dashboard finds nothing there to look at. When a proxy, load balancer or client library cuts a response mid-flight, you do not get a well-formed body with an empty content array; you get a parse error, a connection reset, or a body that ends inside a string literal. The failure is in the delivery of a document rather than in the content of one, which is why its tell sits at the parsing layer instead of in any field.

A complete JSON document with a coherent usage object is proof the response reached you intact, whatever it happens to contain, and the reason is a property of the format rather than a courtesy of the API. JSON has no valid prefix: a document cut anywhere before its final brace leaves an object unclosed, and a parser reports that rather than handing you a partial value. A body that parses at all therefore reached its last byte. The usage object is emitted with the completion rather than ahead of it, so finding it present and internally consistent puts the end of the document in your hands and settles the delivery question before any field inside it is read.

The rightmost leaf keeps the diagnosis honest. A length-based finish reason with a small reasoning count and a full content array is an ordinary truncation — the answer was simply longer than the allowance — and the response to it is different: raise the limit, or instruct the model to be briefer, or paginate the work. Reading every length-based stop as a reasoning problem is the mirror-image mistake to ignoring reasoning altogether, and the reasoning count separates the two in one comparison. Where the API reports no reasoning count at all, the substitute is the ratio of visible tokens to output_tokens: an empty content array on a response billed for 4,096 output tokens accounts for itself without any extra field.

Two beliefs send this diagnosis wrong before the payload is ever opened. The first is that an empty response means either the provider is broken or the prompt is at fault, which is tempting because those are the two explanations an application team can name and one of them is the part they control. Three of the four causes above have nothing to do with the prompt's wording and one of them has nothing to do with the model at all, which is why the ninety seconds spent on a captured response comes first. The second belief is that a length-based finish reason means reasoning ate the answer; that is tempting on a reasoning model because it is the interesting explanation and it is frequently right, and the rightmost leaf of the diagram exists precisely because it is sometimes an ordinary long answer meeting a short allowance.

Why a repointed alias moves the distribution#

A model alias is the undated name in a request body — the one that resolves to whichever dated set of weights the provider is serving under that name today — and it is a mutable pointer. Resolution happens at request time and on the provider's side, which is the property that does the damage: the string in your code is stable and the function that string names is not. Nothing in your repository changes when the pointer moves, nothing in your deployment history records it, and the request you send on Wednesday can be answered by different weights from the one you sent on Monday.

Reasoning length is exactly the sort of behaviour that moves when the pointer does. A newer snapshot of the same model family, at the same effort setting and the same prompts, can deliberate substantially more, because deliberating further is one of the ways a newer snapshot gets better at what it was trained on: more intermediate work before committing to an answer is part of what buys the accuracy the release is announcing. The change that improves the model is therefore the same change that lengthens its output, and the two arrive together whether or not anybody downstream wanted the second one.

Querying a week of usage records on either side of one such repoint, on the agent endpoint above, gives the shape of it:

Nothing there is a defect in the new snapshot, and it is still your outage. The limit was sized against a distribution rather than against a promise: 4,096 was chosen because the traffic of the day fitted inside it, and no part of the provider's contract said the traffic of next month would. Pointing at an alias is a standing agreement to accept whatever distribution comes with whatever is being served under that name, so a limit fitted to the current one is a bet renewed silently on every request.

The headroom that bet was running on is worth computing rather than quoting. The old configuration had 4,096 tokens of allowance against a p99 reasoning length of 3,750, and subtracting the second from the first leaves three hundred and forty-six tokens of room. Three hundred and forty-six against an allowance of 4,096 is the 8% of headroom in that sentence, which means one request in a hundred was already coming within a few hundred tokens of the ceiling before anything changed at all. That is enough to look healthy on a dashboard, where a p99 sitting under the limit reads as a p99 with room, and not enough to survive any change in the underlying distribution.

The belief that fails here is that a quantile with visible headroom is a quantile you can size a limit against. It is tempting because that is what quantiles are for, and it holds only while the distribution that produced the quantile is the distribution you will go on serving. A p99 is a summary of requests already answered, so it says nothing whatever about a distribution nobody has sampled yet, and an alias is a standing invitation for that distribution to change. The 3.1% of requests that now fail are the right tail of the distribution after it shifted, and they are the hardest inputs in the traffic, which is the same selection effect that puts truncations on the longest documents in a summarisation workload.

The cost row shows up on an invoice rather than on a dashboard, and it is worth reconciling against the figure an incident report would quote. Spend went from $19.40 per thousand calls to $23.80, a difference of four dollars and forty cents on a base of $19.40, which comes to a little under a quarter. An increase that lands on a dashboard as roughly a fifth of one day's spend is that ratio seen through rounding and through whatever else the traffic did that day, so the two agree about as closely as an invoice and a table ever do. Reasoning tokens are billed as output tokens, so a mean deliberation that roughly doubled adds its former size to the bill again — and the result is a quarter rather than a doubling because deliberation is one component of a call that also carries 6,214 input tokens.

The clamped p95 and p99 in that table are worth reading carefully, because they under-report the problem, and the chain that makes them do it is short enough to follow to the end. Generation stops when the counter reaches the limit, so a request that would have deliberated for 5,000 tokens and a request that would have deliberated for 9,000 are both halted at 4,096 and both recorded at 4,096. Every request that would have gone further than the limit is written down at exactly the limit, which piles the whole of the distribution's right tail onto a single value. The record is not noisy or approximate about those requests; it is the same number repeated, and the number is the configured ceiling.

A quantile computed from records like that is a lower bound rather than an estimate. A p95 answers the question of what value all but the top few observations fall at or below, and once those top observations have all been rewritten to 4,096, the answer is 4,096 however far the true values reached. The quantile has stopped measuring the workload and started measuring the configuration, and it degrades as the problem worsens: the more requests cross the limit, the more of the distribution stacks onto that one value, and the more confidently the dashboard reports the ceiling back to you. That is why both the p95 and the p99 in the table read 4,096, and why neither of them is telling you anything about the new snapshot.

The information that would fix the estimate is not missing from the records; it was never generated. Those tokens do not exist anywhere, because the model stopped before producing them, so no log, no export and no cleverer query recovers them, and the only way to learn how long the deliberation wanted to be is to let a sample of it run to the end. Re-running at a deliberately generous limit — 16,384 for a few hundred requests — puts the ceiling far enough above the distribution that nothing is cut off, and the quantiles read off that run are quantiles of the workload rather than of the configuration. Here they came back at p99 5,400 and a maximum of 7,100.

That measurement is cheap enough not to argue about, and the arithmetic behind the figure is worth doing rather than quoting. Three hundred replayed requests at a mean of 3,100 output tokens each is three hundred multiplied by 3,100, which is a little over nine hundred thousand output tokens for the whole run. Billing is quoted per million, so that run is a little over nine tenths of a million tokens, and multiplying nine tenths of a million by $15.00 per million gives just under fourteen dollars — $14 for the whole run, rounded up. About twenty minutes of wall clock is what three hundred requests take at any sensible concurrency.

The sample has to be drawn from the failing 3% rather than uniformly, and the reason is a property of the quantity being estimated. What you are trying to learn is how far the right tail of the reasoning distribution reaches, and a request that finished comfortably under the old limit contributes nothing to that: replaying it at 16,384 produces the same short deliberation it produced before, and one more observation in a region that is already densely measured. A uniform sample of this traffic is overwhelmingly made of such requests, since the failing share is 3.1%, so a few hundred drawn uniformly would contain roughly ten that had ever approached the limit and would report, correctly and uselessly, that the distribution is fine.

Sampling the failures spends every one of the three hundred observations in the region where the answer actually lives, and it buys that at a stated price: the quantiles that come back describe the hard requests rather than all traffic. That is the right thing to size a limit against and the wrong thing to quote as a description of the workload. Run the replay against a copy of the production configuration with only the limit changed, because effort setting and prompt version both move the reasoning distribution in their own right, so a sample taken with either of them different measures a distribution you are not deploying — and it will be wrong in whichever direction that difference happens to push, which the result itself cannot tell you.

Two alerts that fire on the mechanism#

Raising a limit is not a detector, and a shift like the one in that table can run for days before anything downstream notices. Two standing alerts close the gap, and both read fields that chapter 1 already put in the log line:

- alert: ReasoningShareHigh
  expr: |
    histogram_quantile(0.95,
      rate(llm_reasoning_tokens_bucket[1h])) /
    histogram_quantile(0.95,
      rate(llm_output_tokens_bucket[1h])) > 0.85
  for: 30m
  annotations:
    summary: "p95 reasoning is >85% of billed output on {{ $labels.endpoint }}"

- alert: LengthFinishRate
  expr: |
    sum(rate(llm_finish_reason_total{reason="max_tokens"}[15m]))
      / sum(rate(llm_finish_reason_total[15m])) > 0.005
  for: 15m
  annotations:
    summary: "{{ $value | humanizePercentage }} of completions cut off by the limit"

The two measure different things, and which of them moves first follows from what each one is a ratio of. The reasoning share is computed across all requests, so it responds to the whole distribution moving: every request that now deliberates a little longer lifts the numerator, and the ratio starts climbing on the first hour of traffic the new weights answer. The length-finish rate counts requests that crossed the limit, which is a tail event — it holds at its old value until some part of the distribution has travelled far enough to cross, and until then it is faithfully reporting zero occurrences of something that is already under way.

The second is the direct detector: a length-based finish reason is a truncated answer, and any sustained rate above a fraction of a percent means the endpoint is shipping incomplete work to somebody. The first is the early warning, because the reasoning share climbs before the truncations start — on the repoint above it went from 0.61 to 0.79 in the twelve hours after the switch, while the length-finish rate was still under a tenth of a percent. Both thresholds are workload-specific and worth setting from a fortnight of your own history rather than from these numbers.

That ordering is what the forty hours in the incident below are made of. The share began moving with the first traffic the new weights answered and went on moving as more of the distribution shifted, while the length-finish rate was still describing a tail that had not yet reached the limit and latency was still inside its own threshold. The rule that fires on the whole distribution fires first because the whole distribution is what moves first, and no amount of threshold tuning reverses that sequence.

Latency moved in that table for a reason worth stating, because it explains both why latency is the signal teams usually page on and why it is not the one to page on. Deliberation is generated a piece at a time, in the same stream as the answer and ahead of it, so the interval a caller waits before the first visible character is the interval spent generating tokens they will never see. Doubling the mean reasoning length therefore doubles that wait, and the table shows the correspondence: the mean reasoning count went from 1,180 to 2,540 while p99 latency went from 4.1 s to 9.6 s, both a little over a doubling, and the two factors land close together because one is substantially the cause of the other.

That same correspondence is the argument against alerting on it. Latency is derived: it is a token count multiplied by a generation rate, plus queueing, plus network time, plus whatever else sits between the client and the weights, and every one of those terms moves for reasons that have nothing to do with reasoning. It is downstream of the real change, it is noisy for a dozen unrelated causes, and it crosses its threshold long after the reasoning distribution has moved. An alert on a symptom two steps removed from the cause buys an incident channel that starts by discussing network paths and connection pools.

Raising the limit on its own#

The proposal that follows an incident like that is to double max_tokens from 4,096 to 8,192 and close the ticket, and it deserves a straight answer rather than a reflex, because it is half of what actually shipped. The empty responses do stop, and the endpoint does start answering the hard requests it was failing, which is a real improvement for the 3%.

What it does on its own is convert a visible failure into an invisible one, and the step that makes that happen is worth spelling out. The model has no target length for its deliberation; it continues until it has reached something it is willing to answer from, and on an effort-driven or adaptive model no internal ceiling stops it earlier. The old limit was therefore not shaping those requests, it was interrupting them, and moving the interruption further out does not make them shorter — it lets each one run to the length it was always going to want. The hard requests that were being cut off at 4,096 now run to 5,400 or 7,100 and are billed for all of it, on every such request, indefinitely.

Latency follows the same curve, so a p99 of 9.6 seconds becomes something closer to 17. The requests that previously threw an exception now return a complete, correct, slow, expensive answer, and nothing in the system says anything changed — the error rate falls to zero, the pager goes quiet, and the only remaining trace is a cost line that drifts up by a few percent a month.

The belief to name is that raising the limit fixed it, and it is tempting because every observable available to check it against agrees. The exceptions stop, the error rate returns to zero, the pager stays quiet, and the handler that used to raise now receives text to parse. All of those are true and none of them is evidence about the thing that changed, which is which requests are slow and what they cost. A complete answer that took seventeen seconds and cost twice what it used to looks, to every field the application can inspect, exactly like a complete answer.

The deeper objection is that the 3% was carrying information. It was the signal that the model behind the alias had changed, and it arrived within two days. Silencing it without pinning the snapshot means the next repoint has no symptom at all until someone reconciles an invoice, which by the numbers above is a quarter of the way through the following month.

An empty completion is one of the better-instrumented failures in this area, because the provider tells you exactly what happened in fields that are already in the response. The comparison is always the same: finish reason first, reasoning against the limit second, and the shape of the body before either, to rule out a transport problem that has nothing to do with the model. Everything after that — an uncensored re-measurement, a pinned snapshot, an alert on the reasoning share — exists to make the next distribution shift arrive as a dashboard line rather than as a page.

Everything in this chapter is a single request in isolation: one prompt, one completion, one usage payload. Multi-step tool loops break that framing, because the reasoning generated on step one has to be present in the request that carries step two, and the rules for how it travels are stricter than anything so far.