Batching and scheduling schema edits

A cold window has a fixed shape: a fixed number of invalidated tokens, and a duration set by how long it takes every serving node to see a request and refill its own copy. Neither is a property of how many edits the deploy carried. Three edits shipped separately therefore cost three windows and three edits shipped together cost one, which makes the batching decision arithmetic rather than taste once the per-window cost is a number somebody can read.

The two questions that remain are which edits are allowed to wait for the next departure, and what to do about the class of change that must not wait and must not be made compatibly either. The second is the harder one, because everything chapters 4 and 5 argued for — add the new shape, keep honouring the old one, retire it on a signal — is precisely wrong when the old shape is the thing you are removing.

One window instead of three#

Three pull requests sat open against one tool registry on the same Thursday afternoon. One fixed a spelling mistake in search_shipments. One added held_at_customs to the status enum on list_orders, because the warehouse system had started emitting that state and calls filtering on it were failing validation. One reworded the description of escalate_to_human from "when the customer asks for a person" to "when the customer requests human assistance", which a teammate found clearer. Each was approved by a different reviewer, each was labelled low risk, and the default behaviour of merging when green would have shipped them at 14:20, 15:05 and 16:40.

Three deploys, three cold windows, roughly 600 requests degraded each time at that hour, about $184 and twelve minutes apiece by the measurement from chapter 6. Putting all three behind one release at 03:10, when the endpoint runs about 6 requests per minute, took longer to close in wall-clock terms — around 22 minutes, because fewer requests arrive to refill each serving node's copy — and touched roughly 130 requests rather than 1,800. The cost came in near $40 total, and no request that hit it belonged to a user with a latency target attached, because at 03:10 the traffic is an overnight batch reconciliation job that does not care about time to first token.

Both figures come out of the same product used in chapter 6 — a population of affected requests, an invalidated region, and the difference between the write and read rates — with only the population changing. Three afternoon deploys means three windows of six hundred requests each, which is the eighteen hundred, and three charges of the same size, which is the five hundred and fifty. The trough deploy is the same sum with a different first term: six requests a minute across the twenty-two minutes the window took to close is around a hundred and thirty requests, carrying the same region at the same rate difference, and it lands near forty dollars. Nothing about the region moved, because the edits are identical in both cases. The population fell by roughly a factor of fourteen and the money fell by the same factor, which it has to, since the region and the rate cancel out of the ratio.

The wall-clock lengthening is worth holding onto, because it is the one place where the intuitive move backfires. Shipping at a trough does not shorten the cold window; it lengthens it. What it reduces is the number of requests that fall inside it, and requests are what you pay for and what carry the latency your users notice. If something about your system cares about the duration of the degraded state rather than its population — a health check that alarms on sustained TTFT, a circuit breaker with a rolling window — a trough deploy will trip it where a peak deploy would not, and that is worth checking once rather than discovering at 03:10.

The lengthening is also far smaller than the drop in traffic, and seeing why settles what a cold window actually is. If the window ended once some fixed number of requests had passed through, then cutting the rate by a factor of eight would stretch it by a factor of eight, and covering at six a minute what fifty a minute covered in twelve minutes would take well over an hour. It roughly doubles instead. The window is not a request count. It is the time taken for every serving node in rotation to see one request and fill its own copy, and a node needs one request rather than a share of the traffic, so a thinner stream reaches all of them later without reaching them proportionally later. The requests inside the window and the duration of the window therefore move by quite different factors, and only one of the two is the thing you are billed for.

Which changes wait for the train#

Batching only works if changes can wait, and the honest version of the policy says which ones cannot. The classification that holds up is by what the change fixes rather than by how large its diff is, since diff size correlates with nothing here — a one-character typo and a security removal are both a handful of bytes and they belong at opposite ends.

The middle rows are where judgement lives. The held_at_customs enum was failing perhaps forty calls a day out of 72,000, each of which the model recovered from within a turn, so a week of waiting cost about 280 recoverable failures and saved a cold window — a defensible trade that would invert immediately if the rate were forty an hour. The reasoning that does not survive is "it is only a description", which was the reasoning behind all three of the Thursday pull requests and which stops being about cost at all once the cost is a printed number.

Both sides of that trade are a rate multiplied by a duration, which is what makes them comparable at all. The cost of shipping is a request rate multiplied by a window length, priced per token. The cost of waiting is a failure rate multiplied by the delay, priced in whatever the failure does — forty calls a day for seven days is the two hundred and eighty recoverable failures, and the inversion at forty an hour is arithmetic rather than judgement, because that same week is then several thousand of them. Judgement enters at one place only, which is what a recoverable failure is worth against a dollar, and that is a question a team can answer once and reuse. The failure it prevents is asking the question afresh for each pull request and answering it from how the deploy feels.

A release train — a deploy that goes out on a fixed schedule, with changes queuing for the next departure rather than shipping the moment they are green — needs somewhere for changes to sit, and the artefact that makes it work is the offset map from chapter 2 rendered per pull request. A CI step that prints "this edit moves the prefix from token 3,940; 61,900 tokens invalidated; ≈ $128 and ≈ 11 minutes at current traffic" turns the review conversation from an argument about principle into a number a reviewer can weigh against the reason for the change. Teams that add this generally find the batching policy enforces itself, because nobody argues for shipping a $128 reword alone once the $128 is in the pull request template.

Nothing in that line is new machinery. It is the same three terms, with the region read off the offset map instead of reconstructed from an incident: the tokens behind the edited tool's start offset, the requests that will fall inside a window at the hour the deploy would land, and the difference between the write and read rates. Sixty-one thousand nine hundred tokens across a peak-hour window of six hundred requests, at that rate difference, is where the hundred and twenty-eight comes from, and it is smaller than the earlier figure for the straightforward reason that a smaller region sits behind that tool than behind the one in chapter 6. A rewording and a rename of the same tool print the same figure, since no term in the product depends on what the edit was. Because every term is known before the merge, the number can be printed at review time rather than recovered afterwards, which is the only difference that matters between this and a postmortem.

The migration where compatibility is the thing being removed#

Everything in chapters 4 and 5 argues for the additive path when a parameter changes shape: add the replacement, keep accepting the old field, map it forward, watch the live, retry and replay counters until they go quiet, remove it around day fifty. Applied to a parameter that is a security hole, that playbook produces a system where the hole remains reachable for fifty days, which is the precise opposite of the remediation. The value of an adapter is that it keeps honouring old calls, and old calls are the vulnerability.

So the parameter is removed outright, and the calls already sitting in live conversations and stored transcripts get refused. Refusal has a design, and getting it wrong wastes the window you just paid for.

{
  "type": "tool_result",
  "tool_use_id": "toolu_01F3xQ",
  "is_error": true,
  "content": "The `filter` parameter has been removed and is no longer accepted. Use `where` with a structured object instead, for example: {\"where\": {\"status\": {\"eq\": \"open\"}, \"created_after\": {\"gte\": \"2025-01-01\"}}}. Retrying with `filter` will fail again."
}

Three properties in that payload do the work. It is permanent and says so, which stops the model burning turns on retries of the same shape — an error reading "invalid argument" produces two or three identical attempts before the model gives up, and each attempt is a full request against a prefix that is currently cold. It names the replacement and shows its shape, so the re-plan has a destination; the model reading this has the new schema in the tools block as well, and the error text is what connects the failed call to the right entry in it. And it is an error rather than a silent empty result, because a tool that quietly returns nothing teaches the model that the query found no rows, which produces a confidently wrong answer to the user instead of a corrected call.

The replay path is where a removed hole comes back#

Leaving the replay path alone is how a removed vulnerability returns. A support harness that replays stored transcripts against a permissive fixture, or against a service configured with the old parameter still enabled "so replays work", has recreated the reachable SQL path in an environment that usually holds a copy of production data. The replay has to hit the same refusal, which means some old bug reports genuinely cannot be reproduced any more, and the harness should label that outcome distinctly rather than reporting a clean non-reproduction — chapter 5's false negative, arrived at deliberately this time and worth marking as such. Queued retries get the same treatment, moved to a terminal dead-letter state with the reason recorded, because retrying a job whose payload can never validate is a loop that ends in a retry budget rather than a decision.

Most of what made this an expensive decision was fixed long before the security review, when someone added a parameter that accepts an arbitrary string whose values cannot be enumerated. A filter that took a raw fragment could never be narrowed without a breaking change, so the migration cost was set at design time and paid at remediation time. The design rule that follows is to prefer parameters whose value space you can list, because a closed set can be extended additively — the enum gaining held_at_customs cost a routine window — while an open one can only ever be closed by breaking somebody.

The asymmetry there is worth one more step, because it is what makes the rule predictive rather than a preference. Adding a member to an enum produces a set that contains the old one, so every call written against the old enum still validates, and no caller anywhere — live conversation, queued retry or stored transcript — is refused. That is what "additive" means in the sense chapters 4 and 5 use it, and it is available because the value space had an outside to expand into. A parameter that accepts an arbitrary string has no outside. It already admits everything, so the only direction left is inward, and every step inward refuses calls that used to work. Which of those two situations you will be in years later is decided by the person who first writes the parameter, and it is decided before there is any traffic to make the decision feel important.

The rest is inventory, and the useful version is short. A canonicaliser so library upgrades stop moving bytes, an offset map in CI so every schema edit arrives with its invalidation cost attached, the legacy-usage counters tagged by live, retry and replay so retirement is a signal rather than a date, the per-minute cache and TTFT query so a deploy can be priced the same afternoon, and a release train with a written rule for which classes of change may skip it. Each of those is an afternoon of work and each one converts a question that currently gets argued into a number that gets read.

The tool schema you send is a deployed API whose clients are a model conditioned on its own history and a directory of stored transcripts. Treating the next one you write with the seriousness that implies means naming parameters as though renaming them will cost fifty days, ordering the array as though inserting into the middle will cost the fleet, and asking in the design review what a change to this tool will invalidate — before there is anything cached behind it to lose.