Percentile-derived caps

Every ceiling in this topic so far has been asserted: 25 iterations, 2 million tokens, €5 a task, €20 a user a day. Choosing those numbers is a separate piece of work from building the machinery that enforces them, and it is the piece that decides whether the machinery ever does anything. A cap can be wrong in two directions, and the two are not symmetrical. Set it too low and you find out within a day, because legitimate work fails and support hears about it. Set it too high and you find out never, because nothing changes: the same code paths run, the same money is spent, and a line in a config file makes everyone feel the question has been handled.

This chapter is about deriving a defensible number from data you already have. It uses one month of task-level usage — p50 8,000 tokens, p95 60,000, p99 210,000, with a maximum of 2.4 million, and cost per task at p50 €0.02, p99 €0.70, maximum €46 — and works through what each candidate ceiling would have done to that month, what has to be written down alongside the number, and why one number fitted across all traffic is wrong for every workload in it.

Replaying the month against candidate ceilings#

The useful exercise is to take the same month of tasks and ask, for each candidate ceiling, how many tasks it would have interrupted and how much spend sits above the line. Both numbers come out of one query over the task ledger.

WITH caps(cap) AS (VALUES (60000), (210000), (400000), (1200000), (2400000))
SELECT
  c.cap,
  count(*) FILTER (WHERE t.input_tokens > c.cap)                    AS tasks_over,
  round(100.0 * count(*) FILTER (WHERE t.input_tokens > c.cap)
        / count(*), 3)                                              AS pct_tasks,
  round(100.0 * sum(greatest(t.input_tokens - c.cap, 0))
        / sum(t.input_tokens), 1)                                   AS pct_tokens_above
FROM tasks t CROSS JOIN caps c
WHERE t.started_at >= now() - interval '30 days'
  AND t.segment = 'interactive'
GROUP BY c.cap ORDER BY c.cap;

Over 1.34 million interactive tasks, that produced:

Before reading anything into that table, check that it says what its column headings claim, because the check is one multiplication per row and it is the only thing standing between a derived cap and a made-up one. A candidate placed at the p95 should leave five percent of tasks above it, and five percent of 1.34 million interactive tasks is the 67,000 in the row. The p99 candidate should leave one percent, which is 13,400. The p99.5 candidate leaves half a percent, 6,700, and the p99.9 candidate a tenth of a percent, 1,340. Each count is exactly the complement of its percentile applied to the same denominator, with no row drifting, which is what licenses calling these figures percentiles at all rather than five numbers somebody liked the look of. It also means the "share of tasks" column carries no new information — you can read it off the percentile label — and that the column worth staring at is the next one, which cannot be derived from the percentile and had to be measured.

One thing that column does not say is worth stating before it gets quoted in a planning document. greatest(t.input_tokens - c.cap, 0) sums the tokens above the line, not the tokens of the tasks that crossed it. So pct_tokens_above is the saving you get if a task that reaches the cap is stopped there and everything below the cap is still billed, which is what happens when the consequence is a truncation, a compaction or a halt partway. If the consequence is instead that the task is killed and reissued in a cheaper form, or refused outright, the saving is larger, and if it is that the task is killed and the user immediately runs it again, the saving can be negative. The query measures the ceiling's mechanical effect on the token volume; it does not know what your boundary behaviour is. Choose that behaviour first, then decide which figure the column is giving you.

Reading the two right-hand columns as a pair, rather than one at a time, is what turns the table into a decision. Going up the candidates, the share of tasks touched falls much faster than the share of tokens bounded: from the p95 to the p99.9 the tasks column shrinks by a factor of fifty while the tokens column shrinks only threefold. Divide one by the other and you get how much spend each percent of interrupted tasks buys you, and that quantity climbs steeply as the cap rises. A high cap is therefore the efficient control and the weak one at the same time — efficient because it bounds a great deal of spend per task it disturbs, weak because the total it bounds keeps falling. There is no ranking that resolves that; there is a point where the second consideration stops paying for the first, and locating it is a judgement rather than a calculation.

The bottom row is the proposal that arrives in most design reviews, and the argument for it is that no legitimate workload would ever be rejected. That is true and is the problem. The cap was derived from the largest thing that happened last month, so it is guaranteed to have fired zero times over the data it was fitted to, and there is no reason to expect a different result next month.

There is a sharper way to say why a zero firing rate is disqualifying, and it is the same test worth applying to any control. Ask what you would observe with the cap configured and what you would observe with the line deleted from the config file. Every task runs. Every euro is spent. The logs are identical, the invoice is identical, and no event is emitted anywhere. A control whose observable consequences do not differ between existing and not existing has not been shown to be weak; it has been shown to be absent, and the config entry is a record of an intention rather than a mechanism. A synthetic task that deliberately crosses the ceiling is the only thing that would ever distinguish the two states, which is why one belongs beside a cap chosen this way rather than beside a strict one.

The pct_tokens_above column is the one that turns this from an argument about principle into an argument about money. A ceiling at 400,000 tokens leaves 34% of the month's token volume above the line, which is 34% of spend that the cap would have prevented or deferred, and it does so by intervening in one task in two hundred. That trade — half a percent of tasks touched, a third of spend bounded — is the shape of a control worth having, and it is visible only because the tail is heavy.

Choosing between 400,000 and 1.2 million is then a product question rather than a statistical one, and it turns on the last column. The quarterly reporting run is a real workload with a real customer attached, so a cap that stops it needs a story for that customer before it ships.

What makes the number defensible#

A defensible cap is a sentence with four named parts: the distribution it came from, the percentile, the consequence at the boundary, and the review that keeps it true. Anything missing one of them will be argued about again in six weeks with no new information.

Per-task input is capped at 400,000 tokens, the p99.5 of interactive tasks over the trailing 30 days. At 85% the task compacts its transcript and drops to the cheaper model tier; at 100% it checkpoints and moves to the approval queue rather than being killed. Reviewed monthly; the cap-hit rate is alarmed outside the band 0.2%–1.0% of interactive tasks.

The consequence is the half people skip, and it is what makes the percentile choice survivable. If crossing the ceiling kills the task and shows an error, then every one of those 6,700 tasks a month is a support ticket and the cap will be raised within a fortnight by someone who does not care about the distribution. If crossing it means the work checkpoints and waits for an operator, the same 6,700 tasks become a queue with an owner, and the cap can stay where the data put it. The strictness of the number and the harshness of the consequence trade against each other, and it is usually better to set the number where the data says and soften the consequence than the reverse.

Writing it down where the code can read it keeps the rationale attached to the value:

per_task_input_tokens:
  value: 400000
  derived_from: p99.5 of tasks.input_tokens, segment=interactive, trailing 30d
  computed_on: 2025-03-11
  at_85_percent: [compact_transcript, downgrade_tier]
  at_100_percent: checkpoint_and_queue
  expected_hit_rate: [0.002, 0.010]
  review_by: 2025-04-11
  owner: platform-agents

Two inputs to that derivation deserve care. Excluding known incidents from the distribution you fit to is a required step and an easy one to forget: percentiles computed over a month containing an abuse event are contaminated by it, so the p99.9 you read includes the thing you are trying to prevent — the month above contained a night of nine hundred restarts and a red-team exercise that ran a compounding loop to €365. Fit on a window with those excluded, then check the resulting cap against them to confirm it would have caught them. Fitting to the maximum instead has the opposite property, since the ceiling is then defined by whatever abuse and whatever bugs your worst month happened to contain, and a worse month next quarter raises it again by the same logic: the control ratchets upward and never downward. A safety factor multiplied onto a maximum compounds that error rather than adding margin, which is how caps end up two or three orders of magnitude above anything they could plausibly restrain.

The ratchet in that rule is worth spelling out, because it is what makes the mistake compound rather than merely persist. A maximum over a window is monotone in what the window contains: a new extreme raises it, and nothing that happens can pull it below an extreme already inside the window. So refitting after a bad month raises the cap, and refitting after a quiet month leaves it where the earlier extreme put it — there is no sequence of observations that lowers it while the extreme is still in view. Now apply the safety multiplier. It is being applied to a value that is already the largest thing observed, so it is not adding margin against the unknown, it is multiplying an outlier, and the next refit multiplies again from the raised base. A percentile has none of this behaviour, because a percentile can move down as well as up when the distribution does, which is exactly the property that lets a cap tighten on its own when a workload gets cheaper.

The distribution is a mixture, so fit it per segment#

A single percentile over all traffic is fitted to a mixture of workloads that have nothing to do with each other, and the resulting cap is simultaneously too loose for one and too tight for another. Splitting the same month by segment made that concrete:

The 2.4-million-token task belonged to the third row, where it is unremarkable. Fitting one number across all three gives a ceiling around 400,000, which is 2.7 times too permissive for interactive chat — the segment carrying 78% of the volume and therefore most of the abuse surface — and tight enough to break scheduled reporting. Segment by the thing that determines the shape of the work, which is usually the entry point rather than the customer.

Both of those claims come out of the table by division, and doing the divisions is what turns "the mixture is a problem" into a size. The share column closes — the three segments account for the whole of the traffic with nothing unassigned — so the mixture really is these three and not these three plus something unexamined. The mixed ceiling of 400,000 divided by the 150,000 that interactive chat's own distribution asks for is the factor of 2.7, and the direction matters as much as the size: interactive chat is the segment carrying more than three-quarters of the tasks, so the segment where the cap is loosest is also the one an adversary has the most opportunities to work in. Divide the other way for scheduled reports and the mixed ceiling sits well below their p99, meaning the cap would be tripping on ordinary runs of a workload that is supposed to be large. One number produced both faults at once, and it did so because a percentile of a mixture is a percentile of nothing in particular — it lands between the components, too high for the one that dominates the count and too low for the one that dominates the size.

When the outlier turns out to be a real customer#

The 2.4-million-token task was traced and it was a genuine enterprise workload: a quarterly compliance report across 1,900 documents, run by a customer paying enough to be in the account management deck. That fact changes the consequence and should not change the ceiling.

Raising a global cap to admit one workload gives every other tenant the same allowance, including the ones you have not met, and it prices your protection at the level of your single most unusual customer. The alternatives are all narrower. Put the workload in a batch lane with its own cap, its own queue and no user waiting on it. Record a per-tenant override with an expiry date and an owner, so it is visible in a review rather than buried. Or require operator approval for tasks above the line, which for three runs a quarter is a trivial amount of human work.

There is also a question worth asking before accommodating it at all, which is whether 2.4 million tokens was the right cost for that job. A run over 1,900 documents that re-sends an accumulating transcript will bill several times what the same job costs with per-document isolation and a map-reduce merge. When that workload was rewritten to summarise each document independently against a fixed 3,000-token prompt and merge in a second pass, it came in at 340,000 tokens and €3.90, down from €46. Raising a cap to fit a workload freezes whatever inefficiency the workload contains, and the customer would rather have the version that finishes in four minutes.

Knowing in three months whether the number is still right#

Four things are worth watching, and only one of them is spend.

The cap-hit rate per segment is the primary signal, alarmed in both directions. Below the band, the cap has gone stale relative to a distribution that shifted down or a workload that moved elsewhere, and it has quietly become decorative. Above the band, something changed — often your own deploy, since a prompt revision that adds 4,000 tokens of instructions or a retrieval change that returns eight chunks instead of four moves p99 overnight and starts tripping ceilings on ordinary work.

The ratio of the cap to the current p99 is the drift measure. It was 1.9 the day it was set; if it reaches 4 the distribution has moved under the cap and the control is loosening without anyone editing it, and if it drops below 1.2 you are about to have a bad week. The count and age of overrides matters for the same reason, because overrides are how a well-chosen cap becomes a fiction: ten tenant overrides with no expiry means the real cap is whatever the most permissive override says.

The 1.9 is the cap divided by the p99 it was fitted against — the 400,000 over the 210,000 in the first table — and quoting it as a ratio rather than as two numbers is what makes it a drift measure at all. A cap in a config file changes only when somebody edits it. The p99 underneath it changes every week without anybody doing anything, so the ratio moves on its own, and it moves in the direction that tells you which way the ground shifted. Rising means the workload got cheaper or moved to another segment while the ceiling stayed put, so the control has loosened with no diff to show for it. Falling means ordinary work is climbing towards a line that was set for outliers, and the cap-hit rate is about to follow. This is the only one of the four signals that reports a change in the thing the cap was derived from rather than a change in the cap's effects, which is why it gives warning rather than news.

The fourth is a synthetic — a task you run against production on purpose, with no customer behind it, purely to exercise machinery you need to know still works — that deliberately exceeds each cap on a schedule, asserting that the task was stopped, the event was emitted and the ledger recorded the halt. A daily canary at €0.60 buys the knowledge that the control is wired to something, which is exactly the property the observed-maximum proposal could never have. Controls that fire once a quarter decay silently, and the first real firing is a bad time to discover that the handler throws on a null field.

A percentile-derived cap has an honest limit worth stating plainly: it describes last month. It cannot anticipate a workload you have not shipped, and it is fitted to traffic that was itself produced under whatever ceilings existed at the time. Caps that adapt — thresholds recomputed nightly against a trailing window, or per-tenant baselines that flag deviation from a customer's own history — handle drift well and create a new hazard, since an attacker who ramps slowly trains the baseline upward. Whatever the derivation, keep a fixed absolute ceiling above the adaptive one that no automation can raise.

What separates a real ceiling from a decorative one is not strictness. It is that the number came from a distribution somebody else can recompute, fitted to the segment it governs with known incidents taken out, carrying a consequence mild enough that the cap survives its first week and a review date that forces the question again. A cap chosen that way will fire on real work occasionally, and that is the evidence it exists.

Everything so far has assumed somebody doing something they should not: an attacker, a payload, a restart loop. The same missing ceilings produce a different-looking failure when the spend is entirely legitimate, the customer is delighted, and nobody notices for a month.