A failed permission lookup says nothing about what the caller may read

Every permission-aware retrieval path contains one call that has to succeed before anything else can happen: resolving what the caller may read. That resolution is usually a network call to a service that owns the grant data, and network calls fail. The question this chapter answers is what a retrieval service should do in the moments after such a call times out, when a user is waiting and the service does not know what they are allowed to see.

Two words are worth fixing before the argument, because the rest of the chapter prices one of them. To fail closed is to deny the request when the check cannot be completed, on the grounds that an unanswerable authorization question is not the same as a permissive answer to it; to fail open is to let the request through and hope. Both are policies, both are implementable, and the choice between them is usually made by whoever wrote the exception handler, months earlier, for a different reason.

A failed lookup is an absence, not a small permission set#

The handler that gets written looks like this, and it is written by someone defending against a transient blip:

try:
    grants = grant_client.effective_grants(subject, org, timeout=0.25)
except (TimeoutError, ConnectionError):
    grants = grant_cache.get(subject) or Grants(groups=[])

Both branches of that fallback are a guess, and they guess in opposite directions. The cached branch answers the question "what may this caller read?" with a snapshot of unknown age, which during a long incident means a contractor whose access was revoked minutes ago continues to receive documents. The empty branch answers it with "nothing", which is not a leak but is also not a truthful answer, because it hands the caller an empty result set that is indistinguishable from a genuine no-match — a confusion chapter 4 takes apart in detail.

The property both branches lose is that a failed grant resolution carries no information about the caller's permissions. It is not evidence of narrow access and not evidence that yesterday's access still holds. It is an absence, and the only response that does not invent information is to say so: return an error, with a status the client can distinguish from a bad request, and let the caller retry.

That is worth walking as a causal step rather than accepting as a principle, because the fallback's appeal is that it looks conservative. A timeout is an event in the network between two services. Nothing about it was produced by consulting the grant data, so its occurrence is statistically independent of what the grant data says. A value derived from an event independent of the quantity being measured contains no information about that quantity. Therefore both branches are producing a permission set by a process that never looked at permissions, and the fact that one of them produces a narrow set is a property of the code rather than evidence about the caller.

What fail-closed costs, as a number#

"Fail closed" is easy to agree with in a design review and expensive during an incident, so it is worth pricing before one rather than during it. Consider a retrieval endpoint contracted at 99.95% monthly availability, calling a permission service measured at 99.9%, with the retrieval service's own independent failures observed at 99.97%. Because the call is serial and mandatory, the endpoint's availability cannot exceed the dependency's, and the first two numbers were never compatible.

Every row of that table is the same arithmetic applied to a different availability figure: take the fraction of the month the component is not available, which is one minus its availability, and multiply by the number of minutes in an average month. That is where the first three rows come from, and checking one of them is enough to confirm the month length being used, after which the other two follow.

The fourth row closes with two instruments that share no step, which is what makes it trustworthy rather than merely stated. The note in the right-hand column computes it one way: multiply the two availabilities to get the probability that both components are up, subtract from one to get the probability that at least one is down, and convert to minutes. The left-hand column can be reached a second way: simply add the two budgets, because when either component's failure takes the endpoint down, the outages accumulate. Both routes give the same figure.

They agree because the term they differ by is negligible, and naming that residual is part of understanding the row. Adding the budgets counts twice the time when both components are down at once; the product form counts it once. That overlap is the product of two small fractions of a month, which works out at well under a second — far below the rounding in the table. The agreement is therefore a real check on the arithmetic and not a coincidence, and it would stop being a check if either dependency were much less available, since the overlap term grows with the product.

That is the honest answer to what fail-closed costs: about 35 minutes a month of contractual breach, at whatever the SLA credit is — the refund the service-level agreement promises for missing its own number — plus the reputational cost of the incidents being visible. Read the last row as a gap and as a ratio at once: the overrun is the ceiling minus the contract, and the multiple is the ceiling divided by the contract, and both are properties of the two availability figures alone. No amount of care in the retrieval service moves them, because its own contribution is already the smallest of the three.

Stating it that way changes the conversation, because the problem stops being a disagreement about principles and becomes an architecture problem with known solutions. You cannot put a 99.9% dependency in the serial path of a 99.95% endpoint and meet the contract, whatever the failure policy is, so the fix has to remove the dependency from the serial path rather than change what happens when it fails.

There are three ways to do that, and all of them are more work than an exception handler. The first is to replicate the grant data into the retrieval service's own storage — a materialized view of the grant graph, meaning a stored copy of the answers rather than a query that recomputes them, kept current by a change stream of the writes the permission database emits, and read locally in under a millisecond. The dependency becomes the replication pipeline rather than a synchronous RPC, and the availability question turns into a staleness question with a measurable lag, which chapter 6 shows how to hold to a number. The second is to carry a small, short-lived authorization token minted by the permission service at session start, so the grant resolution happens once per session rather than once per query; the floor on revocation latency then equals the token lifetime, which is a deliberate, written-down number instead of an accident. The third is to narrow what the synchronous call has to answer: resolving "which groups is this subject in?" requires walking a graph, while resolving "does this subject have a direct grant on resource R?" is a point lookup, and a degraded path that only serves directly-owned resources needs only the second.

The three are not variations on one idea, and it is worth seeing what each one actually changes. The first removes the call. The second reduces how often the call happens. The third reduces what the call has to compute. Each therefore trades against a different quantity — storage and replication lag, revocation floor, and answerable scope — which is why a system can adopt more than one of them without the benefits overlapping.

The proposal that always follows, and why its bound is fictional#

Within a day of an incident like this, someone proposes serving from the last cached grant set whenever the permission service is unavailable, with a short TTL — the time-to-live after which a cached entry counts as expired, and five minutes is the number people reach for — so that the exposure window is bounded and the endpoint stays up. It is a genuinely reasonable-sounding trade, and the bound is fictional in exactly the situation it exists to cover.

A TTL evicts an entry so that the next request can fetch a fresh one. During the incident, fetching is what is broken, so eviction does not produce a fresh entry; it produces a cache miss against an unavailable dependency. Every implementation of this pattern therefore resolves the conflict in one of two ways, and it is worth checking which one your cache library chose, because the docs often bury it. Either the stale entry is retained past its TTL until a refresh succeeds — the stale-while-revalidate shape — in which case the five-minute bound is really "five minutes, or the duration of the incident, whichever is longer, and the incident is why we are here." Or the entry is genuinely evicted, in which case the request now has no grants at all and the system falls back to the empty-grants branch, which is fail-closed with worse ergonomics and a delay before anyone notices.

That is the stumble, and it is a belief that a defence works. The TTL is a constant, written into a config file, that appears to bound exposure. What it actually bounds is how long an entry survives given that refreshes succeed, and whether refreshes succeed is precisely the property that varies during the event the bound exists for. The number stays five minutes while the thing it is supposed to limit runs for as long as the outage does. It is tempting because in every ordinary minute of the service's life the bound is real and observed; it only stops holding in the minutes nobody tests.

The second problem is an ordering inversion that gets worse under attack. The revocations most likely to be issued during an incident are the ones that matter most: a compromised account being cut off, a contractor terminated at short notice, a customer invoking a data-deletion request. Serve-stale honours those last, because they are precisely the changes that have not propagated. An attacker who can cause or predict permission-service degradation gets a window in which revocation is disabled, which converts an availability weakness into an authorization bypass. That is not a hypothetical property of the design; it is what the design says it does.

If it ships anyway, argue for these bounds#

Degraded modes exist because businesses have to keep running, and refusing to design one usually means someone builds a worse one under pressure. The version worth arguing for constrains scope and evidence rather than time alone.

Deny any resource whose grant changed inside the stale window. This requires a second, much cheaper feed than the permission service itself: an append-only log of recently-changed grant subjects and resources, which the retrieval service tails and keeps for at least the maximum stale window. During degraded mode, any resource appearing in that log is excluded, whatever the cached grant set says, which converts the risk from "all revocations are ignored" to "recent changes are treated as denials." The feed has to be independently available — if it lives in the same service and fails with it, the correct behaviour is to deny everything, because an unavailable change log means you cannot tell which grants moved.

That feed is doing something the TTL could not, and the difference is worth naming. A TTL reasons about time since a snapshot, which is a proxy for whether the snapshot is still true. The change log reasons about whether anything relevant moved, which is the quantity itself. Substituting a measurement for a proxy is what turns the bound from fictional into real, and it works during the outage precisely because the log is a different service with a different failure mode.

Narrow the scope to resources the caller owns outright. Group-derived and inherited grants are where staleness hurts, since those are the ones that change when someone leaves a team or a folder is reshared. A direct grant from subject to resource is both cheap to verify and rarely the thing being revoked in a hurry. In the system above, restricting degraded mode to directly-granted resources cut the addressable corpus for a typical caller from about 62,000 chunks to about 4,100 and still answered a little over half of the queries received during the window.

Those two figures are worth reading as a gap rather than as a level, because the gap is the whole justification for the narrowing. The addressable material fell by well over an order of magnitude while the share of answerable queries fell by a little under half. Queries are therefore concentrated on directly-owned material far more heavily than the corpus is, which is what makes the restriction a good trade rather than an arbitrary one — and it is a measurement of that system's query mix, not a general law, so it is the thing to check before adopting the bound elsewhere.

Cap the mode by wall clock, behind an explicit break-glass, with an audit trail. A break-glass control is one that cannot be reached in the ordinary course of operations and leaves an unmistakable record when someone reaches for it, named after the fire alarm you have to break to pull. Degraded mode should not activate itself on a timeout; it should be switched on by a named human via a flag that expires automatically after a fixed interval — 30 minutes is long enough to be useful and short enough that renewing it is a decision someone makes on purpose. Every response served in that mode carries a flag, every request is logged with the age of the grant set used, and the aggregate is reportable afterwards: how many requests, for which subjects, against how stale a snapshot. That record is what turns a period of degradation into something you can hand a customer.

def resolve_or_fail(subject: str, org: str) -> Grants:
    try:
        return grant_client.effective_grants(subject, org, timeout=0.25)
    except (TimeoutError, ConnectionError):
        pass

    mode = breakglass.state()            # off | on(expires_at, opened_by)
    if not mode.active:
        raise PermissionsUnavailable()   # -> 503, Retry-After

    snapshot = grant_cache.get(subject)
    if snapshot is None or not change_feed.healthy():
        raise PermissionsUnavailable()

    grants = snapshot.direct_only()      # drop group-derived grants
    grants = grants.exclude(change_feed.resources_changed_since(snapshot.at))
    audit.degraded_read(subject, snapshot.age_s, mode.opened_by, len(grants))
    return grants

Note what is missing from that function: a TTL. The bound on staleness is the age recorded in the audit line and the exclusion set from the change feed, and both of those keep working while the permission service does not. Note also the order of the two guards. The break-glass check comes before the cache read, so a service that has not been switched into degraded mode never consults a snapshot at all, and the healthy-feed check comes before the snapshot is used, so an unavailable change log denies rather than degrades.

Which failure you would rather explain#

The ordering is not a matter of taste, and it holds for a structural reason. An outage is bounded, symmetric and reversible: it affects everyone equally, it ends when the dependency recovers, its cost is written into the contract as a credit, and after it ends the system is in the same state it was before. A cross-tenant disclosure is unbounded in time, asymmetric in who it harms, and cannot be undone — the customer's document is in another customer's possession, and no amount of subsequent engineering retrieves it. It also converts an engineering problem into a legal one, since most jurisdictions and most enterprise contracts treat unauthorized disclosure of customer data as a notifiable event with a clock measured in hours.

The asymmetry that decides it is that availability is purchasable and confidentiality is not. Thirty-five minutes of overrun is a roadmap item with known fixes — replicate the grant data, shorten the serial path, add a bulkhead that keeps one dependency's failure from consuming the whole service's capacity — and each of those fixes is a quarter of work with a predictable outcome. There is no corresponding project that un-discloses a board minute.

That asymmetry is also why the pricing exercise above is not an argument for failing open at some price. The two costs are in different units: one is minutes against a credit schedule, the other is an event with a notification clock and no upper bound on consequence. Comparing them requires a judgement rather than arithmetic, and the point of putting a number on the availability side is to stop the conversation being only a judgement, not to reduce the other side to a number as well.

Fail-closed leaves one thing unresolved, and it is the thing the empty-grants branch tripped over. Returning an error is a decision about what the retrieval layer does; it says nothing yet about what the caller is told, or about what happens when the permission lookup succeeds and the answer is genuinely that this caller may read nothing at all. Those two situations produce the same zero passages, and only one of them is safe to answer from.