Timestamp precision and placement

Every value at the top of a prompt is a bet about how many requests will share it. The policy behind it is a good bet — it changes when somebody edits it, which is weekly at most. A timestamp rendered to the second is the worst available bet, because it is unique to the request that produced it, and prefix matching halts at the first token that differs. The tokens behind it are as unreachable as if they had been moved to the end of the prompt.

That makes timestamp handling two decisions rather than one: what precision the prompt needs, and where in the prompt that value sits. They are separable, they have different answers for different rules inside the same policy, and both are settled by reading what consumes the value rather than by argument. The measurements below come from a scheduling assistant handling about 40 requests an hour against a five-minute cache lifetime, on the common hosted pricing shape of a write at 1.25× the base input rate and a read at 0.1×.

Eleven tokens in front of eight thousand#

That assistant reports a cache hit rate of 0.0%. Not four percent, not a bad afternoon — zero, on every request, for as long as the panel has existed. Its prompt is 8,000 tokens of policy about meeting types, room booking rules, notice periods and cancellation windows, followed by the user's request and whatever the calendar tools returned. The policy has not been edited in five weeks. Every block in the prompt is stable, the ordering follows the stable-first rule from chapter 2, and the write counter reads 8,014 on every single call.

The system block begins:

Current date and time: 2026-08-07T14:03:11Z
You are the scheduling assistant for Northwind. Meetings of type...

Those first eleven tokens are different on every request, so the 8,000 tokens behind them cannot be reused. Follow the consequence rather than accepting it. The match halts at the first differing token, so it halts inside the timestamp, which means the longest prefix any two of this surface's requests share is a handful of tokens — shorter than any provider will cache. So every request finds no usable entry, and finding no entry is not the same as doing nothing: the cache_control marker is still on the block, so the request writes a fresh entry, and a write costs 1.25× the base rate. The bill is therefore the whole prompt at 1.25× instead of the whole prompt at 1.0×, which is 25% above what this service would pay with caching switched off entirely. The write counter reading 8,014 on every call is that sentence stated as a measurement.

What a bucket is worth at your arrival rate#

The counter-proposal that lands in most review threads is to keep the timestamp at the front and truncate it to the minute, so that every request arriving inside the same minute shares a prefix and hits. That is a real idea with a computable answer rather than a mistake, and the answer depends entirely on how many requests land in a bucket.

Every rollover abandons a warm entry and pays for a new one, so the question is how many reads a bucket collects before it rolls. Take the simplest case first. Two requests land in the same minute. The first writes the prefix at 1.25× and the second reads it at 0.1×, so the pair costs 1.35 units of prefix between them. Send the same two requests with no caching at all and they cost 1.0 each, which is 2.0. Caching wins, and it wins by a lot, which is why the idea keeps being proposed.

The general form of that comparison is one line of algebra. For N requests in a bucket, one of them writes and the other N−1 read, so the cached cost is 1.25 + 0.1(N-1) units and the uncached cost is N. Setting the two equal finds the N at which caching stops helping. Expand the left side to 1.25 + 0.1N - 0.1, which is 1.15 + 0.1N. Subtract 0.1N from both sides — allowed because it is the same quantity taken from each — and what is left says that 1.15 equals nine tenths of N. Divide both sides by nine tenths and the crossover is at N of about 1.28. Below that many requests per bucket the cached layout costs more; above it, less. So the break-even is not two requests per bucket but a little over one, and dismissing bucketing out of hand is as sloppy as proposing it.

What makes it a loss in practice is that arrivals are not evenly spaced, and the gap between "the mean is above 1.28" and "most buckets are above 1.28" is where the idea fails. Requests arrive independently at a mean rate of λ per bucket, which makes the count in a given bucket Poisson, and the probability that a Poisson count is zero is e^-λ. So the fraction of buckets that contain at least one request is 1 - e^-λ. Each of those buckets contains exactly one write, no matter how many requests it holds, because the first request writes and the rest read. Writes per bucket is therefore 1 - e^-λ while requests per bucket is λ, and dividing the first by the second gives the fraction of requests that pay a write: (1 - e^-λ) / λ. At 40 requests an hour, λ is 40 divided by 60, or 0.67 per minute, and the formula returns 73% — so nearly three requests in four are the only one in their minute and pay the premium, even though the mean of 0.67 is a number a spreadsheet would happily average away.

The "effective cost per prefix token" column is that write fraction turned into a price. For the per-minute row, 73% of requests write at 1.25× and the other 27% read at 0.1×, so the average is 73% of 1.25 plus 27% of 0.1, which is 0.937 — against 1.0 for sending the prefix uncached, hence the −6% beside it. Run the same two lines for the to-the-second row, where every request writes and none reads, and the average is the write rate itself, 1.25, which is the +25%. Run them for the hourly row, where λ is 40 requests per bucket and almost every request finds a warm entry, and the average collapses towards the read rate itself.

Per-minute bucketing on this surface saves 6% of the prefix cost. It also adds a rotating cache key, a clock dependency in the prompt assembler, a new failure mode when two application servers disagree about the current minute by a few hundred milliseconds, and a hit-rate panel that will read around 27% forever and look broken to whoever inherits it. Six percent does not buy that.

The money is also not the whole bill. A read against a warm entry skips the prefill for those tokens — prefill being the pass in which the model reads the whole input before it emits anything — which on this deployment is worth about 240 ms of time to first token on 8,000 tokens, so a layout where 73% of requests write rather than read leaves p95 latency, the figure 95% of requests come in under, almost exactly where it was before caching existed. Bucketing at a granularity that produces a 27% hit rate buys 6% of the tokens and roughly 6% of the latency benefit, and a team that adopted it would still be looking at a slow endpoint while believing the caching work was done.

The last two rows are equal for a reason worth stating plainly: once a bucket is long enough that rollovers are rare compared to the TTL, the bucket stops being the binding constraint and the TTL takes over. A daily rotation costs one write a day; the entry it writes will have idled out long before the day is over, so what actually decides the write count is how often the traffic goes quiet for five minutes, and that is the same whether the prefix rotates daily or never.

The scheduling assistant's requests average 90 seconds apart, comfortably inside a five-minute window, but averages are again the wrong instrument. A measurement over one week's logs found 63 gaps longer than five minutes per weekday, mostly early morning and around lunch, and each of those is an entry that expired and a request that has to write a new one. Put that against the day's traffic — 40 requests an hour over 24 hours is 960 — and 63 of the 960 write while the rest read, so the effective cost per prefix token is 63 writes at 1.25× plus the remaining requests at 0.1×, divided by 960, which comes out near 0.175 rather than the 0.101 the table's idealised row shows. Bucket granularity sets a ceiling on reuse and the TTL sets the floor, and on a low-traffic surface the floor is what you are actually standing on.

Deciding what precision the timestamp needs#

The precision question is answerable by experiment rather than by argument, because it is a question about which rules in the policy consume the time value. Read the policy for every rule that references time, then ask what resolution each one needs. This policy has fourteen such rules, and the audit fits in a table:

Eleven of the fourteen need day precision or coarser and can read a date that rotates once a day. Three of them need the clock, and all three are answering a question about the turn in front of them rather than applying a standing rule, which is the pattern that keeps recurring: material a rule consumes belongs with the rule, and material a specific answer consumes belongs with the specific answer.

The audit is doing something more specific than sorting rules into two piles, and it is worth naming because it is the step that converts a design argument into a measurement. The precision a prompt needs is not a property of the prompt and not a matter of taste. It is the maximum over the precisions its individual consumers require, so a single rule that needs the minute forces the whole value to the minute if there is only one value. Splitting the value into two — a coarse one in the prefix and a fine one in the tail — removes the maximum by removing the "only one value" premise, and each half is then free to sit at the precision and the position its own consumers want. That is why reading the policy for consumers, rather than debating what precision "feels safe", is what ends the argument.

def build_request(user_msg, tools, now, tz):
    system = [
        # cacheable prefix: nothing here varies within a day
        {"type": "text",
         "text": f"Today is {now.astimezone(tz):%A, %d %B %Y} ({tz}).\n" + POLICY,
         "cache_control": {"type": "ephemeral"}},
    ]
    messages = [
        *retrieved_calendar_blocks(user_msg),
        # volatile tail: exact clock time lives here, next to the turn that needs it
        {"role": "user", "content":
            f"<current_time>{now:%H:%M:%S %Z}</current_time>\n{user_msg}"},
    ]
    return {"system": system, "messages": messages, "tools": tools}

Putting the date in the prefix at day precision is still a rotation, and it is a cheap one: one invalidation per day per timezone, at a moment you can predict. The clock time moves to the last position in the prompt, which is where a value the model needs for arithmetic in this specific turn belongs anyway, and which costs eleven uncached tokens per request. Price that before deciding it is acceptable rather than after. Eleven tokens on each of 40 requests an hour for 24 hours is 40 × 24 × 11 ≈ 10,600 tokens a day, which at this model's input rate is about three cents. Set the two sides of the trade beside each other: three cents a day of permanently uncached tail against 8,000 tokens per request of permanently uncached prefix. What makes the tail cheap is that eleven tokens are the whole of what has to leave the prefix to fix it.

Whether the coarser date still supports the rules that consume it is a question the eval answers rather than the arithmetic. Running 400 paired cases against both builds moved the notice-period rule from 99.1% to 99.0%, which is noise, and left the quarter-freeze rule unchanged. The three clock rules were the ones that needed watching: a build that dropped the timestamp entirely, on the theory that the model could infer the time from the calendar tool results, answered "starts in eleven minutes" style questions incorrectly in 6.2% of cases, usually by rounding to the nearest half hour. Eleven tokens in the tail took that back to 0.4%.

The fix that fragments the prefix instead of killing it#

The build above formats the date in the user's timezone, which is the natural thing to write and which quietly replaces one failure with a subtler one. Northwind's staff sit in 38 timezones, and the string Today is Friday, 07 August 2026 (Australia/Sydney) differs from Today is Friday, 07 August 2026 (Europe/Madrid) in the tokens that carry the zone name, so the prefix forks 38 ways. At 40 requests an hour spread across those zones, each variant serves about one request an hour against a five-minute entry lifetime. An entry that lives five minutes and is used once an hour is nearly always cold when it is wanted, so mean reuse inside the window is around 0.4 and the hit rate lands near 11% rather than the 82% the daily rotation promised.

That failure is a different mechanism from the one at the top of the chapter, and the difference is worth holding because the repairs differ. In the opening case the prefix changed on every request, so there was no reuse to have. Here the prefix is perfectly stable — for each timezone — and the problem is that the traffic has been divided 38 ways, so each version of it is read too rarely to survive between uses. One is a stability failure and the other is a concentration failure. Both land on a dashboard as a low hit rate, which is why the dashboard cannot tell you which repair to make.

The repair is to keep the prefix canonical and let the tail carry anything user-specific: Today is Friday, 07 August 2026 (UTC) in the cached block, the user's zone and local time in the same eleven-token tail element that already carries the clock. That restores a single prefix shared by every user in the world for a day at a time, and it costs six more tokens per request.

def first_divergence(request_a, request_b):
    """Where two prompts stop sharing a prefix, in characters and tokens."""
    a, b = serialize(request_a), serialize(request_b)
    limit = min(len(a), len(b))
    offset = next((i for i in range(limit) if a[i] != b[i]), limit)
    return {"char_offset": offset, "approx_tokens": offset // 4, "context": a[max(0, offset - 40):offset + 40]}

# scheduling assistant, two requests 27 seconds apart, before the fix
# {'char_offset': 42, 'approx_tokens': 10, 'context': 'Current date and time: 2026-08-07T14:03:11Z\nYou are the sc'}

Two requests that ought to share a prefix, serialized and compared, give the offset directly, and a number under the provider's minimum means the mechanism is not running at all rather than running badly. Wiring that probe into an hourly job against sampled production traffic is what caught the timezone fork: the expected offset was somewhere past 8,000 tokens and the job reported 1,510, four days before the monthly invoice would have said anything.

A third option is worth knowing about for agents that already have tools: expose get_current_time and let the model call it when a turn actually needs the clock. That keeps the prompt entirely static and moves the timestamp into a tool result, at the cost of an extra round trip on the turns that need it — around 900 ms on this deployment — which is a bad trade for a surface where most turns need the time and a good one where few do.

Where a rotating prefix is the right design#

Rotation is a legitimate pattern when the rotating content is genuinely shared. A product catalogue regenerated every Monday, a per-tenant policy block that changes when the tenant edits it, a digest of open incidents injected into an ops assistant — each of these produces a prefix that many requests share for hours or days, and the cost of the rotation is one write per period. The ops assistant is the tightest of the three and it still works comfortably: the digest regenerates every fifteen minutes, the surface takes about 60 requests a minute, so each version of the prefix is read roughly 900 times before it is replaced, and the write premium is amortised over all of them at a cost of about 0.1% of the period's prefix spend. The pathology in the scheduling assistant is not that the prefix changes; it is that it changes faster than it is read.

That gives a sharper test than "keep the prefix stable". For every value at the front of a prompt, ask how many requests will see that exact value before it changes. If the answer is a number in the thousands, it belongs in the prefix. If it is one, it belongs in the tail. If it is somewhere in between, the bucketing arithmetic above tells you what the middle is worth, and it is usually less than the moving part costs to own.

Precision and position answer to the same test. Coarsening a value raises the number of requests that share it, and moving a value behind the breakpoint takes it out of the count altogether — so the date goes in the prefix at day resolution because thousands of requests see that string, and the clock goes in the tail because one request does. Everything in between is a bucketing decision whose value you can compute before you build it.

Whether the model still books rooms correctly after the date has moved out of the policy block and into a differently-shaped sentence is a separate question, and on this fix the eval answered it. It usually is not asked. The next chapter is about what happens three weeks after a layout change ships on the strength of a cost graph alone.