A revocation SLA is a number you measure, not a shrug at consistency

Revoking someone's access is a write, and the write is the fast part. An administrator removes a user from a group, a row changes in a grant store, and the transaction commits in a few milliseconds. What a compliance commitment promises is something else entirely: that every reader has stopped honouring the previous answer. Those two events are separated by every copy of the grant data that exists between the write and the query — replicas, caches, tokens, and requests that were already in flight when the write landed.

"Eventual consistency" is the correct name for that separation and it is not a measurement of it. The phrase says the copies converge; it says nothing about when. A commitment of the form revocations take effect within five minutes is a claim about the largest gap between the commit and the last reader still acting on the old grant, and it is meaningful only once somebody has measured the interval it names. The usual first attempt at that measurement is a load test that races a revocation against a query, and such a test returns a real number that describes a subset of the system, because which copies it can observe is decided by how the test itself was configured.

Defining the quantity precisely enough to measure#

The number that matters is the revocation propagation time: the interval between a grant change committing and the completion of the last request anywhere in the system that can still act on the previous grant. Two details in that sentence do most of the work. Measuring to completion rather than to the start of the request is what makes it an exposure window rather than a scheduling statistic, because a request that resolved grants 5 ms before the commit and streams its answer for another 8 seconds has exposed the document for 8 seconds. And anywhere in the system includes paths that are not the primary query path, which is where the surprises live.

Both details follow from asking what the number is for. It exists to bound the period during which a person who should no longer see a document can still see one. A request that has already been authorized and is still writing bytes to a client is showing that person the document, so the exposure has not ended when the authorization check ended; it ends when the last byte lands. And a person does not care which component supplied the document to them, so a bound that covers the main query path and omits a cached response is not a bound on the quantity at all — it is a bound on one route to it.

The window decomposes into contributors that can be measured separately, which is what makes it manageable. In one production system carrying a five-minute revocation commitment, the accounting looked like this:

The column that carries the argument is the last one, and the useful reading is the gap between it and the two columns to its left. Every row's bound is a configured quantity — a TTL somebody chose, a token lifetime somebody chose, a request deadline somebody chose — while the typical and p99 columns are observed quantities. Where those two diverge sharply, the observed number is telling you how often the condition holds and the bound is telling you what happens when it does, and only the bound belongs in a contract.

Three rows carry a dash in the typical column, and that dash means something specific rather than "not measured". Those three contributors are absent from an ordinary request and appear only when a particular condition holds: a pod-local cache has an entry for this subject, a response cache has an entry for this query, or the caller is presenting a token minted before the change. A contributor that is usually zero and occasionally enormous is exactly the shape that a percentile taken over ordinary traffic will hide, and it is also the shape that a load test will miss unless the test is built to create the condition.

How the contributors combine is the part the table compresses, and getting it wrong in either direction gives a wrong bound. Two contributors that lie in series on one request's path add: a pod cache holding stale grants until its TTL expires, followed by a request that started just before the expiry and then streamed its answer to completion, exposes the document for the cache interval plus the streaming interval. Two contributors that are alternative routes to the same staleness do not add: a request served from a response cache and a request authorized from a stale pod cache are two different ways for one request to be wrong, and a given request takes one of them. So the end-to-end figure is the maximum, over the possible paths, of the sum along each path — not the sum of the column and not the largest single entry.

Running that rule against the observed maximum leaves a residual worth naming rather than smoothing over. Half a minute of pod cache followed by a little over eight seconds of streaming, plus the replication and code-path terms which together are under a fifth of a second at p99, comes to a couple of seconds short of the stated end-to-end maximum. The table does not say where the remainder comes from. Two rows carry bounds far larger than the observed maximum, so neither of them was live on the path that produced it, and nothing in the accounting says which configuration made them inert. A decomposition that accounts for most of a measurement and not all of it is still the right instrument; presenting it as complete is what would be wrong.

The diagram is worth drawing once per system, because the arrows nobody thinks about are the ones between the commit and a component that holds a copy. Every such component contributes a term, and the end-to-end number is the largest live one rather than the sum, since they overlap. Read that alongside the series rule above: the terms overlap when they are alternative routes, which is the common case across components, and they compose when one component's staleness hands off to another's — which is exactly what the diagram draws, a cached authorization at the start of a request whose streaming carries it past the moment the cache would have expired.

Measuring it continuously rather than once#

A propagation time established by a load test in March is a claim about March. The measurement that survives an audit is a loop that runs in production, forever, against real infrastructure, and reports a distribution. It works on a canary: a purpose-built identity and a purpose-built document that exist only to be granted, revoked and searched for, so that the probe can prove a leak with a string nothing else in the corpus contains.

def probe() -> float:
    """Returns seconds from revoke-commit to last observed stale read."""
    subject, group, doc = canary_identity(), CANARY_GROUP, CANARY_DOC

    grant_store.grant(subject, group)
    wait_until(lambda: can_read(subject, doc), timeout=120)   # confirm baseline

    committed_at = grant_store.revoke(subject, group)         # returns commit ts
    last_stale = None
    deadline = committed_at + timedelta(seconds=600)

    while now() < deadline:
        resp = query_as(subject, CANARY_QUERY)                # full public path
        if CANARY_DOC_ID in [c.doc_id for c in resp.citations]:
            last_stale = resp.completed_at
        elif last_stale and now() - last_stale > timedelta(seconds=60):
            break                                            # 60s clean, call it
        sleep(0.25)

    return (last_stale - committed_at).total_seconds() if last_stale else 0.0

Three properties of that loop are the ones that make it evidence rather than decoration. It goes through the public path, with a real token obtained the way a user's token is obtained, so it exercises the token row of the table rather than skipping it. It records the time of the last stale read rather than the first clean one, so a system that flickers — replica A caught up, replica B did not — reports the tail rather than the median. And it keeps probing for a full minute after the last stale hit before declaring success, because a leak that reappears at t+45 s from a cold cache is exactly the failure this exists to catch.

The wait_until call before the revoke is doing quiet work and is easy to delete as ceremony. It confirms that the canary can read the document before the revocation, which is what makes a subsequent clean read informative. Without it, a probe that reports zero seconds is ambiguous between "propagation was instant" and "the grant never took effect, so there was nothing to revoke", and those two are the same observation from the outside. Establishing the baseline is what converts an absence into evidence.

Running the probe every 90 seconds produces around 950 samples a day. That figure closes against the cadence with nothing else in it: ninety seconds is forty intervals in an hour, and forty an hour across twenty-four hours is the stated count. What it licenses is a statement about which statistics from that stream are trustworthy over which horizon. Roughly a hundredth of a day's samples sit in the tail a p99 is computed from, which is about ten observations daily, so a p99 stabilises within a few days and can be alerted on. A maximum is a single order statistic drawn from the extreme of the distribution, and a single unlucky day can move it, which is why it is quoted over the longer window rather than over the last one.

What you publish from that is a percentile and a maximum, not a mean, because the commitment is a bound: p50 1.1 s, p99 9.8 s, max over 90 days 41 s, against a written SLO of p99 ≤ 30 s and max ≤ 300 s — the service-level objective, which is the internal target you hold yourself to, set tighter than the customer-facing agreement so that missing it is a warning rather than a breach. The maximum is the number the contract is about; the p99 is the number that tells you whether the maximum is a fluke or a trend.

The alert has to fire on the observed tail, not on the mechanics. Alerting on "the revocation job completed" is the most common version of this mistake, and it is what makes a dashboard green during an incident, because the job finishing says nothing about whether the read path has noticed. A pod whose grant cache is not receiving invalidation broadcasts will serve stale grants indefinitely while every job in the pipeline reports success, and the probe is the only thing that sees it, since the probe is the only component that reads the way a user reads.

That mistake is worth naming as a belief rather than as an oversight, because it is a defence people trust. Job completion is tempting as a signal for two good reasons: the job is the thing that was actually built, and it reports truthfully about itself — when it says it finished, it finished. What makes it useless here is that its output is a quantity that stays green while the dangerous property varies completely. Whether any reader still holds the old grant can go from zero readers to every pod in the fleet without the job's status changing by a character, because a pod that has stopped consuming invalidation messages is not a participant in the job and is polled by nothing. An indicator that cannot move when the hazard moves is not a weak indicator; it is measuring a different thing.

Shortening the window, and what each technique actually bounds#

Once the contributors are enumerated, three techniques address them, and they bound different things.

Cache TTL is the weakest, because it converts an unbounded window into a bounded one without shortening the typical case. A 30-second TTL means the worst case is 30 seconds and the average is 15, and lowering it to 5 seconds multiplies grant-store read load by six. Both halves of that follow from one fact about a cache with a fixed lifetime. A request arriving at a uniformly random moment within an entry's life sees an entry of average age half the lifetime, which gives the average; the worst case is a request arriving just before expiry, which gives the TTL. And the rate of refetches is one per entry per lifetime, so it varies inversely with the lifetime — cutting the TTL to a sixth multiplies the read rate by six, exactly, with the entry count cancelling out of both sides. That inverse relation is why a TTL is the right tool for components you do not control well enough to invalidate and the wrong tool for shortening a typical case.

Active invalidation broadcasts the change — a pub/sub message, published once and delivered to every subscriber, that each retrieval pod consumes on revocation and applies to its local cache — and typically brings the observed window down to tens of milliseconds. What it does not do is bound anything, because a pod that missed the message is stale until its TTL expires, and messages get missed during exactly the network events that make you care. Broadcast plus a short TTL is the combination that gives both a good typical case and a real ceiling; broadcast alone gives you a good typical case and a dashboard that says everything is fine.

The two compose cleanly because they act on different statistics of the same distribution. Broadcast moves the median and leaves the tail where the TTL put it; the TTL moves the tail and leaves the median where the broadcast put it. Neither substitutes for the other, and a team that ships one and reports the statistic the other governs has published a number its system does not hold.

Fencing is the technique that makes the bound structural rather than statistical. Each grant change increments a monotonic version, meaning a counter that only ever moves upwards so that any two values can be ordered, requests carry the version of the grant snapshot they authorized against, and the read path refuses to serve a response authorized against a version older than some threshold — either by re-resolving or by failing. The word doing the work there is structural. Under a TTL the ceiling holds only so long as each component honours its own expiry, so a component that misbehaves raises the ceiling silently and nothing reports it; under fencing the check sits on the response path itself, and an authorization that is too old is refused wherever it came from, by a component that has no stake in the caches upstream. This is more machinery than most systems need, and it is what makes it possible to state a bound that does not depend on every cache behaving. The cheap approximation is a hard deadline on the request: if an authorization decision may be at most 2 seconds old when a response completes, then the in-flight row of the table is bounded by 2 seconds regardless of how long the model wants to stream.

The token row usually cannot be fixed inside the retrieval service, which is why it deserves a separate conversation with whoever owns identity. The options are short access-token lifetimes with silent refresh, or removing grant claims from the token entirely so that authorization data always comes from the store, at the cost of the synchronous dependency chapter 3 priced. Choosing the second and then adding a local replica of the grant data is the combination most systems converge on.

A stated bound is a disclosure, and freshness is a different quantity#

A revocation SLA is not a statement that no unauthorized read occurs. It says the opposite: that unauthorized reads are possible for up to N seconds after a revocation, and that N is known. That is a disclosure, and it belongs in the same document as the commitment, because a customer who reads "revocations take effect within five minutes" and assumes it means "instantly" has been given a number without being told what it measures.

It is also a distinct quantity from index freshness, which teams often merge into one dashboard. How long after a document is uploaded it becomes retrievable is an availability question, and a laxer answer — twenty minutes — is usually fine. How long after a grant is revoked the document stops being retrievable is a confidentiality question, and the two should not share an SLO, a dashboard or an alert threshold even though they travel through some of the same pipeline. The reason to keep them apart is that their errors are not symmetric in the same direction: a slow ingest makes a document temporarily invisible to people entitled to it, which is an inconvenience that ends, while a slow revocation makes a document temporarily visible to someone who is not, which is a disclosure that does not.

For an auditor, the evidence package is the written SLO, the probe's retained time series with its maximum, the alert configuration showing what fires and to whom, and one more thing that is more convincing than any of them: a retrospective query over real revocations. For each grant revoked in the last 90 days, take the commit timestamp, then search the access log for reads of affected resources by that subject afterwards, and report the largest gap. In the system above that was 12,400 revocations, a p99 of 8.2 seconds and a maximum of 41 seconds, which matched the probe closely enough to demonstrate that the probe measures the real system.

That match is doing more work than the sentence describing it suggests, and it is worth spelling out. The probe's maximum over its retained window and the retrospective query's maximum over twelve thousand four hundred real revocations are the same figure, and the two instruments share no step: one synthesises a revocation of a canary grant and watches for the document with a string nothing else contains, while the other reads an access log after the fact for revocations that happened anyway, with no synthetic identity and no purpose-built document involved. Two independent routes to one number is what licenses the claim that the probe measures the system rather than measuring itself. Without the retrospective, a probe's output is consistent with a probe that exercises only the paths its own setup creates, and there is no way to tell the difference from inside the probe.

The p99 figures from the two instruments do not match as closely as the maxima, and that is the expected direction rather than a defect. The probe fires on a fixed cadence and samples the system whenever it happens to fire; the retrospective sees whatever real revocations occurred, whose timing correlates with human activity and therefore with load. The maximum is a property of the system's worst path and both instruments eventually find it; the middle of the distribution is a property of when each instrument looked.

All of this measures the paths you enumerated. The enumeration is the fragile step: every technique in this chapter takes the contributor table as given, and a component that holds a copy of the grant data without appearing in that table contributes a term nobody bounds and nobody probes. The test suite is the thing that is supposed to catch the paths you did not, and a suite asserting that tenant A never retrieves tenant B's documents can stay green through the incident it was written to prevent.