The thinking budget

Reasoning tokens are billed at the output rate and their count is decided by the model rather than by you, which makes a cap on them the first control anyone reaches for. Most APIs offer one: an integer budget, expressed in tokens, that bounds how far the model may deliberate before it starts writing. What that integer means depends on where the tokens are drawn from, and on a large family of providers they are drawn from the same allowance as the answer — the max_tokens you were already setting.

This chapter is about the arithmetic of that shared pool: what the two settings jointly guarantee, why a configuration that looks generous fails on a specific and valuable slice of traffic, what the API can catch for you and what it cannot, and how to size both numbers from data you already have. The failures here are quiet by construction, because a truncated answer is a well-formed HTTP success with a field nobody reads.

One pool, and a ceiling on part of it#

On a provider whose max_tokens covers thinking plus visible output, a budget and an output limit are not independent limits on two separate things. They are one pool and a ceiling on part of it. A budget of 6,000 says the model may spend up to 6,000 tokens deliberating; a max_tokens of 8,192 says the whole completion — deliberation and answer together — stops at 8,192. What that configuration guarantees the answer is the difference: 8,192 minus 6,000, which is 2,192 tokens. That is a guarantee rather than an expectation because it is what remains when deliberation runs all the way to its ceiling, which is the worst case for the answer and the only case worth sizing against.

Why that ceiling is not a reservation is visible in how the tokens are produced. A model answering with reasoning enabled emits one continuous stream of pieces — the deliberation first, the answer after it, one at a time, with no gap between the two where anything could be set aside. A counter runs over that stream from the first piece to the last, and max_tokens is the value that counter is compared against; when the comparison succeeds, generation stops, wherever in the stream the model happens to be. A thinking budget is a second counter over the deliberation portion of the same stream, and when it reaches its own value the model is forced out of deliberating and into answering. Nothing there puts tokens aside for anyone: there is no allocation step, only two comparisons.

Four properties follow from that arrangement directly, and each can be checked against it rather than taken on trust. Unused budget costs nothing, because a comparison that never succeeds leaves no mark on what was generated. The answer is guaranteed only the difference, because the worst case for it is the one where deliberation ran to its own ceiling first. There is no warning at the boundary, because neither counter is visible to the model. And a budget cannot make a model think harder, because raising a value the deliberation counter never reaches changes no comparison that was going to succeed.

The belief that gets substituted for that arrangement has a reason behind it. A budget, everywhere else the word appears, is money set aside: allocating it to one line reduces what the other lines may draw on. Reading thinking.budget_tokens that way gives the sentence a reader confidently writes after one pass — that the budget reserves capacity for thinking and leaves the answer's allowance alone. On a provider that meters reasoning separately the sentence is true, which is why it survives contact with experience, and chapter 3 covers that arrangement. On a shared allowance it is false, and the two counters are the reason.

The arithmetic that gets a 6,000-against-8,192 configuration approved uses the typical reasoning length in place of the budget. Those are different numbers, and the gap between them is the whole problem. A budget is a ceiling on how far the model may go, not a statement of how far it does go, and the model decides per request. On an easy input it might spend 900 tokens thinking. On a hard one it spends everything it is allowed.

That substitution is harder to catch than a mistake about a definition, because arithmetic feels settled once it has been done. A sum built on the typical reasoning length is true of the typical request, silent about every other one, and comfortably clear of the limit, which is what a reviewer in a hurry is looking for. The sum that governs the failure uses the ceiling instead, because the ceiling is what the hard requests actually spend.

The right-hand column is that second sum written out: 6,000 of deliberation plus a summary that wants 2,600 is 8,600, which is 408 more than the 8,192 the pool holds. The summary therefore stops after the 2,192 tokens left for it, 408 tokens short of where it was going to end. The reader is handed an argument with its conclusion missing rather than a preamble, and the direction is what matters when somebody asks how bad a given truncation is.

The two columns are not two independent draws, because one property of the input drives both. A long or intricate document gives the model more constraints to reconcile — more clauses qualifying each other, more places where an earlier definition changes what a later term means — and reconciling them takes more deliberation. The same document gives the model more material that has to appear in the answer, since a summary of a lease with schedules has to say something about the schedules. Reasoning length and answer length are two measurements of one underlying difficulty rather than two rolls of a die, which is why the requests where reasoning approaches its ceiling are disproportionately the requests where the answer is long as well.

That correlation decides the shape of the failure. Were the two lengths uncorrelated, an under-sized configuration would fail only when a long answer happened to land behind a long deliberation, which would be uncommon and would pick its victims at random — a low background rate of odd responses spread evenly across document types, which gets attributed to the model and lives in a backlog. Because the two move together, it fails on the slice where both are large, and that slice already has a name in the business: long leases, master services agreements, anything with schedules attached. The complaint therefore arrives pre-sorted by document type rather than as a scattering of odd responses, which is also why it arrives at all.

There is no warning at the boundary, because nothing reads either counter back to the model. It is not told that it has 2,192 tokens left for prose; it generates until a counter reaches 8,192 and then the response is terminated wherever it happens to be. If reasoning consumed 6,000 and the answer needed 2,600, you get the first 2,192 tokens of the answer and a length-based finish reason. If reasoning consumed all 8,192 on its own — possible when the budget is set at or near max_tokens — you get no visible content at all.

What reaches a user depends entirely on how the client handles that. A client that streams output into a panel and renders whatever arrives shows a truncated summary as a summary that happened to end early, and an empty one as a rendering bug. The response carries the evidence the whole time, in a stop_reason of max_tokens, and the general shape of this failure is that the API reports it correctly and the application discards the report. Any client that renders model output without branching on the finish reason will present a cut-off answer as a complete one, and the reader has no way to tell the difference — a paragraph that stops mid-clause reads as an editorial choice more often than it reads as a defect.

Two beliefs meet at that point, and both are reasonable enough that naming them is worth more than blaming the client author. The first is that a response which parses, carries a success status and contains prose is a complete answer, which holds for almost every API an engineer has integrated and fails here because completeness is reported in a field rather than in the status line. The second is that a truncation will announce itself to whoever reads the output, so a check for it would be redundant — which is true of a program's output and not of a summary, where the ending has no expected shape to be missing from. Neither belief is careless, and together they are why a stop_reason can sit unread in every response for weeks.

The version that fails loudly#

The budget-as-allocation misreading has a benign form, which is worth seeing because it teaches the semantics in one error message. It is what anybody configuring a drafting endpoint for the first time runs into: set max_tokens to 1024 and a thinking budget of 2000 on a shared-pool provider, and read what comes back.

client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    thinking={"type": "enabled", "budget_tokens": 2000},
    messages=[{"role": "user", "content": brief}],
)

{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "thinking.budget_tokens: must be less than max_tokens"
  },
  "request_id": "req_01HZ4t9kQm"
}

That request never reached a model. No tokens were generated, nothing was billed, and the failure arrived in milliseconds, because the configuration is contradictory on its face: it authorises the model to spend 2,000 tokens out of an allowance of 1,024. The API can see the contradiction from the two integers alone, so it rejects it at validation. Implementations of this regime commonly enforce a floor on the budget itself as well — 1,024 tokens is a typical value, though it is a provider's choice rather than anything the shared pool implies — on the grounds that a few hundred tokens of deliberation is worse than none, and reject anything smaller with a similar message.

The instructive part is what the API cannot see. max_tokens 8,192 with a budget of 6,000 is a perfectly legal configuration; there is nothing contradictory about it, and there are workloads for which it is exactly right. Whether it is right for yours depends on the distribution of your visible output lengths, which is a fact about your traffic that no validator has access to.

Two kinds of wrongness are at work there, and the difference between them accounts for the difference in when each is discovered. A budget above max_tokens is a contradiction inside the request: everything needed to detect it arrived in the same body, so the check is a comparison of two integers on the way in. A budget of 6,000 under a max_tokens of 8,192 is a mismatch between the request and the world: the request is internally consistent, and what makes it wrong is a distribution of document lengths living in a system the provider has never seen. No care on the provider's side closes that second gap, because the information that would close it was never sent. The loud failure is the one the provider can catch for you. The quiet one is yours.

Sizing it so the tail survives#

The rule that follows from the pool model is mechanical: size max_tokens to the reasoning ceiling plus the worst-case answer you are willing to serve, not the typical one.

max_tokens  >=  thinking_budget  +  p99(visible_output_tokens)  +  margin

The p99 in that expression is the term people skip, because it is the one requiring data rather than judgement. It is usually available without an experiment: an endpoint that retains its own answers — which most do, for support and audit reasons — was retaining them before reasoning was enabled, so tokenise a month of them and read the quantile off the distribution. On a contract-summarisation endpoint, answer lengths ran 800 tokens at the median, 1,900 at p95 and 2,600 at p99, with a maximum of 3,400 on a lease with eleven schedules.

Use p99 rather than the maximum, and the reason is a trade with both sides visible. A ceiling is paid for on every request, in the worst-case cost you have to quote and the worst-case latency you have to design timeouts around, so sizing to 3,400 rather than 2,600 raises the ceiling on everything the endpoint will ever serve to accommodate one lease that turned up once in a month. Sizing to p99 declines that trade and takes on an obligation in exchange: the remaining one percent will be handled by the truncation path rather than avoided. That commits you to having a truncation path — a client that branches on the finish reason, says the answer was cut short and offers a retry — because without one, choosing p99 is choosing to serve the top percent of documents silently damaged.

With a 6,000-token budget and a p99 visible summary of 2,600 tokens, the floor is the sum of the two: 6,000 plus 2,600 is 8,600, so 9,216 or 10,000 is the honest setting, both above the floor with room left for the margin term. The worst-case cost per call follows from the ceiling, because the ceiling is the most output a single call can bill. 10,000 tokens is a hundredth of a million, so at $15.00 per million the worst case is a hundredth of $15.00, which is 15.0 cents. The 8,192-token configuration is a little over four fifths of that ceiling, so its worst case is a little over four fifths of 15.0 cents, which is 12.3. Those figures are what a finance conversation needs rather than the average, since the average is a claim about traffic mix that will change. The alternative move is to attack the other term: cut the budget to 4,096 and keep max_tokens at 8,192, which leaves 4,096 tokens of guaranteed headroom for prose and caps the worst-case cost of a single call where it already was.

Both headroom figures there are the same subtraction on different pairs: raising the limit leaves 10,000 minus 6,000, which is 4,000, and cutting the budget leaves 8,192 minus 4,096, which is 4,096. The two options buy the answer almost identical guarantees, which is what makes the choice between them a question about the other rows.

The latency row is worth reading off the ceilings rather than accepting. Tokens are generated one after another and latency is set by how many are actually generated, so the longest a configuration can take is proportional to the most tokens it permits. The two ceilings, 10,000 and 8,192, stand in a ratio of roughly six to five, and the two worst-case latencies, ~180 s and ~150 s, stand in that same ratio.

Neither column is free, and the second is the one people reach for without testing, because cutting a budget feels like tightening a limit rather than changing a behaviour. It is a behaviour change, and the mechanism is in how the model reaches its answer. Deliberation is text the model writes and then reads back — intermediate results, a clause resolved one way and then another, each piece available to every piece generated after it. Cutting the budget from 6,000 to 4,096 removes about a third of that text on the documents that were using all of it, and so about a third of the intermediate results the answer gets written from. The documents using all of it are, by the correlation above, the hard ones.

That is why the last row of the table differs between the columns. Raising the limit changes what the endpoint spends and how long it takes, and both already have a number beside them on a dashboard. Cutting the budget changes what the model concludes, which no dashboard records, so whether it costs you accuracy is an empirical question with an answer specific to your task. Run a held-out set — inputs with agreed-correct answers, kept out of whatever tuning produced the configuration — at both budgets before shipping the cheaper one, or you have traded a visible failure for an invisible quality regression.

What a budget reserves, and what it steers#

A budget does not reserve anything, and the two-counter arrangement is what shows up on the bill. A 6,000-token budget on a request where the model deliberates for 900 tokens bills 900 tokens of reasoning, not 6,000, and leaves the other 5,100 available to the answer. The deliberation counter reached 900 and was compared against 6,000 without the comparison ever succeeding, and a comparison that does not succeed leaves no trace in what was generated. Unused budget is not spent, not charged, and not wasted; the ceiling costs you nothing until the model reaches for it. That is why raising a budget is a cheap experiment on average and an expensive one at the tail.

It is not a floor either, so it will not make a model think harder about a problem it finds easy. The same comparison explains why: raising a value that the deliberation counter never reaches changes no comparison that was ever going to succeed, so the generated stream is identical token for token. Doubling the budget on a workload that never approaches the current ceiling changes nothing at all — not cost, not latency, not accuracy — and teams occasionally ship exactly that change and then report that reasoning budgets do nothing. They do nothing when the binding constraint is elsewhere.

It is also not a quality dial with a linear response. Accuracy against a budget tends to rise steeply and then flatten, and on some tasks it turns over, because a model given a large budget on an under-specified question will explore branches that make the final answer worse rather than better. The budget is worth sweeping on your own evaluation set — 1,024, 2,048, 4,096, 8,192 — and the useful output of that sweep is the point where the curve flattens, which is the setting to ship.

And it is not a latency limit, though the two move together. Latency is set by the tokens actually generated, so a budget bounds the worst case rather than the common one; a request that deliberates for 900 tokens is fast regardless of what the ceiling says. If your requirement is a hard deadline rather than a token cap, the budget is the wrong instrument and a client-side timeout is the right one.

Two integers, then, and only one relationship between them worth remembering: on a shared allowance the budget is subtracted from the answer's room, in the worst case, on the requests where the worst case is most likely. Size the pool from the budget plus a measured p99 of your visible output, sweep the budget on a held-out set rather than guessing at it, and branch on the finish reason so that the day the sizing is wrong you hear about it from a dashboard rather than from a customer six weeks later.

All of this rests on an assumption stated once and not defended since: that thinking and visible output are drawn from the same allowance. That is one of the arrangements on offer, and the one where the arithmetic above is even meaningful. Ship the same two settings to a provider that meters reasoning separately and the contradictory configuration is accepted rather than rejected, the guaranteed-headroom subtraction has nothing to subtract, and the bill goes somewhere the reviewer was not looking.