Per-server sampling limits

The one control most clients implement for sampling is a clamp on maxTokens, the field a server sets to say how much output it would like. Clamping it is cheap, it is visible in the request handler, and it is easy to write down in an integration guide as the sampling budget control. It is also a bound on a single dimension of a single call, which is a different quantity from what an integration costs you over an hour.

Least privilege — the principle that a component is given exactly the access its job requires, for as long as it requires it, and nothing beyond — is the frame worth applying here, because the protocol offers none of it. The client declares support for sampling once, in initialize, as a boolean: "sampling": {} or the key omitted. Every connected server sees the same declaration, so at the protocol level the grant is binary and undifferentiated. Nothing in that handshake says which servers, how often, on which models, with what context, or up to what spend. Treating a permission of that shape as least privilege is the same category error as granting a cloud identity role s3:*, which is every action on every storage bucket in the account, on the grounds that the action namespace exists.

The cap bounds a call, and the bill is a sum over calls#

Take a concrete request and follow the multiplication. A continuous-integration triage server asks the model to read a stack trace from a failing test and name a likely cause, returning a small JSON object with a cause, a confidence and a file. That is a modest, genuinely useful request, and close to the archetype sampling was designed for. Each call carries roughly 2,900 input tokens — the trace, plus the source file the client attached — and returns about 700 output tokens. At $3 per million input tokens and $15 per million output, that is a shade under two cents a call.

Capping the output of one request bounds one quantity, and that quantity is worth computing rather than accepting, because every later figure in this chapter is a multiple of it. Prices are quoted per million tokens, so each half of the call is a count divided by a million and multiplied by a price. The 2,900 input tokens at three dollars a million come to a shade under nine tenths of a cent. The 700 output tokens at fifteen dollars a million come to a little over one cent. Their sum is the figure above, and the interesting part is how unlike each other the two halves are: the output is under a quarter of the tokens and more than half the money, because a token leaving the model is priced at five times a token entering it.

Nothing about the size of one call is the problem. A build that fails 1,340 tests samples 1,340 times, because the server samples once per failure. It also retries, up to twice, whenever the completion does not parse as the JSON object it expects — and truncated output does not parse, so a 2,048-token clamp is itself producing retries on the longest traces. At an average of 1.5 attempts per failure, one run issues about 2,010 requests, and two runs overlapping on different branches during one hour come to 4,020 requests and $77. Every one of those requests is inside a 2,048-token policy. Four thousand requests an hour is not an abuse of the rule; it is inside it, in the same way that a speed limit is silent on how far you drive.

Follow that multiplication rather than trusting it, because its shape is the argument. One run is 1,340 failures multiplied by an average of one and a half attempts each, which lands a little above two thousand requests. Two runs overlapped inside the same hour doubles it to a little above four thousand. Four thousand requests at just under two cents apiece is seventy-seven dollars. Every term in that product except the price is a count of something — failures, then attempts per failure, then concurrent runs — and maxTokens appears in none of them. The clamp fixes one factor in a product of four, which is why an integration can stay inside it and still return a bill nobody recognises.

The clamp is also weaker than it looks on a single request. It bounds output only. Input tokens come from the messages the server wrote and from whatever context the client attached under the ceiling established in the previous chapter, and neither of those is what maxTokens describes. In the triage traffic the input half was 2,900 tokens against 700 out, so more than four fifths of every request's tokens were outside the one control the team believed they had. A server that pastes 60,000 tokens of log into messages and asks for a 50-token verdict is fully compliant with a 2,048-token clamp and costs nine times what the triage server did.

Both of those ratios come out of the same two figures and they are not the same ratio, which is a distinction worth holding onto whenever anyone quotes a percentage here. Four fifths is a share of tokens: 2,900 in against 700 out. Measured in money the split reverses, because the two token kinds carry different prices, and the input half of a triage call is under half its cost. The log-pasting server is the same sum with the counts swapped. Sixty thousand input tokens at three dollars a million is eighteen cents, its fifty output tokens contribute well under a tenth of a cent, and eighteen cents against just under two is the ratio quoted above. Changing the denominator from tokens to dollars without saying so is the commonest way a spend discussion goes wrong, and the clamp sits on the axis that is cheap per unit and free to grow.

There is a further wrinkle in that maxTokens arrives as a field the server sets. Treating the server's own number as your limit means the limit is whatever the server asks for; clamping it downward is the client's decision, and it interacts with server behaviour in ways worth predicting. Clamping aggressively truncates completions, truncated completions fail server-side validation, and a server that retries on validation failure converts your ceiling into extra requests. A 2,048-token clamp is, on the worst traces, a spend amplifier.

The specification is not silent on any of this. Its sampling section recommends rate limiting and human approval in the same breath as it defines the message shape, and it does so without specifying a mechanism, because the mechanism depends on what your application knows about the servers it has connected. That division is deliberate and it is easy to misread as an omission. The protocol tells you what a request looks like on the wire; whether it is answered is left to you, and a client that answers everything has not implemented the recommendation, it has skipped it.

Seven controls that fail in different directions#

What bounds a server is a contract your client holds about that server, evaluated on every request. Seven controls do most of the work, and they fail in different directions, which is why the set matters more than any member of it.

The consent row deserves the most care, because it is the control teams reach for first and the one that decays fastest. A prompt shown to a human on the first sampling request of a session is a real decision, made once, with attention available. The same prompt shown on every request from a server that samples once per failing test is 1,340 dialogs, and the observed behaviour under that load is a user who holds down the approve key. Consent granularity is therefore a design parameter with a cost curve, not a slider to be set to maximum: per-session consent that names the server, the model tier, the context scope and a spend ceiling teaches more than per-request consent that shows a wall of server-authored text at machine speed.

Read down the second and third columns and the argument for applying them per server rather than in aggregate becomes concrete. A fleet-wide rate limit of 200 requests per minute means one triage server can consume all of it and every other integration on every other user's session sees throttling it did not cause. A fleet-wide spend budget has the same defect on the money axis. Per-server bounds make the blast radius of a misbehaving integration equal to that integration, which is the property you actually wanted from a limit.

Where the policy lives and in what order it runs#

The policy is a record per server, resolved at connection time and consulted on every sampling/createMessage. Keeping it as data rather than as conditionals scattered through the request handler is what makes it reviewable, diffable, and answerable when someone asks what a given integration is allowed to do.

# sampling-policy.yaml — one block per registered server
servers:
  ci-triage:
    allow_sampling: true
    consent: per_session          # first request of a session asks the user
    models: [claude-haiku-4-5]    # cheap tier only; hints outside this are ignored
    max_context_scope: none       # nothing from the conversation, ever
    max_output_tokens: 1024
    max_input_tokens: 8000        # the half maxTokens does not cover
    rate_limit:
      requests_per_minute: 20
      burst: 40
    budget:
      window: 24h
      usd: 15.00
    deadline_ms: 20000
    max_concurrent: 2

  wiki-search:
    allow_sampling: false         # the default for anything not deliberately enabled

The evaluation order is not cosmetic, because each check costs something different to run and each one that fails should fail before you have spent anything.

def authorize(req: SamplingRequest, pol: ServerPolicy, state: ServerState) -> Decision:
    if not pol.allow_sampling:
        return Decision.deny("policy.sampling_disabled")

    if state.concurrent >= pol.max_concurrent:
        return Decision.throttle("policy.concurrency", retry_after_ms=2_000)

    if not state.rate_limiter.try_acquire():
        return Decision.throttle("policy.rate_limit", retry_after_ms=state.next_slot_ms())

    # Context is resolved before pricing, because it changes the input count.
    ctx = resolve_context(req.include_context, pol, state.session, req.server_id)
    model = pick_model(req.model_preferences, pol.models)      # advisory in, allowlist out

    est_in = count_tokens(req.messages, req.system_prompt, ctx)
    est_out = min(req.max_tokens or pol.max_output_tokens, pol.max_output_tokens)
    if est_in > pol.max_input_tokens:
        return Decision.deny("policy.input_too_large", observed=est_in)

    est_cost = price(model, est_in, est_out)
    if state.spend_24h + est_cost > pol.budget_usd:
        return Decision.deny("policy.budget_exhausted",
                             spent=state.spend_24h, cap=pol.budget_usd)

    if pol.consent == "per_request" or state.first_request_of_session:
        return Decision.needs_consent(model=model, est_cost=est_cost, context=ctx.manifest())

    return Decision.allow(model=model, context=ctx, deadline_ms=pol.deadline_ms)

Two details in that function are easy to leave out and expensive to leave out. The estimate is computed before the call and charged against the budget optimistically, then reconciled against the actual usage the provider reports, because a budget enforced only on settled cost lets a burst of concurrent requests all pass the check and overshoot together. And the model comes back from pick_model as the intersection of the server's advisory preferences with your allowlist, never as the server's hint directly; the response still reports whatever ran, which is the field a well-written server reads to decide whether the answer is worth using.

A rate limit is not a defence against intent#

These controls bound carelessness well and hostility only partially, and the distinction matters when someone reads the policy file and concludes the problem is solved. An adversary who wants your context and your money will sit at 19 requests per minute and spend $14.90 a day, indefinitely, and every one of those requests will be authorised. A rate limit converts a fast loss into a slow one, which is worth a great deal operationally and nothing at all as a boundary. What decides whether that slow drip matters is the context ceiling from the previous chapter, because the request that costs three cents and takes nothing is merely expensive while the one that takes a connection string is an incident.

The two figures in that sentence are not independently reachable, and finding out which of them binds is the same arithmetic that makes a policy block legible. A daily budget of fifteen dollars, at just under two cents a triage-sized call, buys about seven hundred and eighty requests, which is also roughly one request every two minutes averaged across a day. The rate limit of twenty a minute permits nearly thirty thousand in the same day. The budget is therefore the tighter of the two by a factor in the tens, and the adversary can have the burst or the day but not both: nineteen requests a minute exhausts a fifteen-dollar budget in about forty minutes. The general rule is that for any window, the binding control is whichever one implies the smallest request count over that window — concurrency for the instant, the rate limit for the minute, the budget for the day — and a set of limits that all bind at the same scale has redundant members in it.

Reading the Tuesday numbers the same way is what makes them trustworthy in the first place. Fifty-seven thousand requests at just under two cents apiece is about eleven hundred dollars, and the invoice said $1,090: two instruments, one a count the client kept and the other a total the provider computed, landing on the same number. That agreement is what licenses everything the platform team concluded from it. It says the sampling traffic is essentially the whole of the excess rather than one contributor among several, so the controls that bound sampling are the controls that would have bounded the day. Without the agreement the table of counts would be a hypothesis, and the correct response to a tripled invoice that no count explains is to go and find the unexplained requests before writing any policy at all.

The budget is also not the provider's rate limit, and confusing the two is common. Your provider enforces limits per API key across every caller sharing it, so when one server exhausts them the throttling lands on your own application's foreground requests — the user's chat, not the server's triage. Per-server budgets exist partly to keep an integration from consuming the capacity your product depends on, which is an availability property rather than a cost one and does not show up on an invoice at all.

The deadline is the control most often left out, and its absence shows up as an availability problem rather than a cost one. A sampling request that the client forwards to a provider and then waits on indefinitely holds a slot in whatever concurrency budget the client runs, and a server that opens eight of them during a provider slowdown has parked eight slots for as long as the slowdown lasts. Twenty seconds is a defensible ceiling for a triage-sized completion and it is short enough that a stuck request resolves into a decision rather than into a hang. What the client does when that deadline fires is a separate question from whether it fires, and answering it badly is how a bounded request turns into a blocked agent.

Nor does any of this tell you which server spent the money after the fact. The policy in this chapter is enforcement: it decides, in the moment, whether a request proceeds. Reconstructing what happened over a month, across a fleet, when the only ground truth is a provider invoice, is a different problem and needs different machinery.

A single number on a single request was never going to bound an integration, because the quantity anyone cares about is spent over a window and an integration is a loop. The seven controls above are worth holding as a set for the same reason: each one has a column of things it does not touch, and the ones a given server will find are not predictable in advance. What makes the set enforceable rather than aspirational is that it lives per server, as data, evaluated in an order where the cheap refusals come first and nothing is spent before the expensive checks run.

The immediate gap is smaller and closer. authorize returns denials and throttles, and so far they exist only inside the client's process. A server that has sent a request is still waiting on a response, and what it receives — a JSON-RPC error it can classify, a timeout it cannot, or a silence it will retry through — determines whether it degrades sensibly or hammers the same request until something gives.