Rotation, audit and kill switch are all-or-nothing

The audit was routine and the number looked good. Over 30 days, 94.1% of the company's model requests had gone through the gateway: 41.2 million requests carrying 8.9 billion input tokens, every one of them attributed to a principal and a feature. The remaining 5.9% was a single workload, a nightly enrichment job that read the previous day's support tickets, summarised them, and wrote the summaries back to the warehouse. It had been written four months before the gateway existed, it called the provider directly, and it held a provider key issued to the company account. Its owning team's response to the finding was that 94% was good enough for now and they would migrate in the next quarter, after a database migration they had already committed to.

That is a reasonable-sounding position and it is wrong, though not for the reason usually given in the meeting. The objection is not that 6% is a lot of traffic. It is that three of the gateway's guarantees are not percentages at all.

Some properties average and some conjoin#

Anyone who has argued about where to put a constraint has met this distinction. A NOT NULL enforced by the database holds no matter which application writes; the same rule enforced by every application's validation code holds only if every application has it, and the ninth service written by the team that joined last year decides whether the invariant is true. The rule does not become 94% true. It becomes false, and you find out when a null shows up in a report.

Three of the things a gateway is bought for behave exactly like the database constraint.

Rotating a compromised credential is the clearest. The enrichment job's key is issued against the same provider organisation as the gateway's key. If a vendor-side compromise or a broad exposure forces you to rotate everything in that organisation, the job's key is in scope, and rotating it means the coordination described in the previous chapter — find the team, stage the new value, confirm the cutover, delete the old key — for a job that only runs at 02:00 and can only be verified the following night. The gateway's promise was that containing an incident is one revocation. With one holder outside, the promise is that containing an incident is one revocation and a rotation whose timeline you do not control, which is the same operational shape as having no gateway at all, just with fewer parties.

A complete audit trail is the second. "Complete" is the operative word, because the value of the trail is what it lets you assert. If a customer asks whether any of their support ticket text was ever sent to a model, or a regulator asks for the retention posture of every prompt containing personal data, the answer you can give from a 94% trail is that you found no evidence in the paths you record. The enrichment job processes support tickets, which is to say it processes precisely the category of data the question is usually about. Over a quarter, the missing records are roughly 4.1 million requests, each one a ticket body, and no query you run over the gateway's logs will mention them.

Blocking a model in an emergency is the third. When the direction is "stop sending customer data to vendor X by end of day", a config change at the gateway stops 94.1% of it, and the job runs at 02:00 with its own key and its own SDK and does the thing you have just told the business you stopped doing. The failure is worse than not having a kill switch, because you told people the switch worked.

The bottom half of that table is why the "good enough for now" position is tempting rather than absurd. Most of what a gateway does really does improve smoothly with coverage: every request you move onto it is a request whose cost you can see, whose latency you can chart, and whose team quota it counts against. If cost visibility were the whole point, 94% would be a genuinely good number and next quarter would be a fine time to finish. The three rows at the top are the ones that make the deadline non-negotiable, and naming exactly those three is the difference between a mandate people accept and a mandate people route around.

It is worth being precise about the overstated version of this argument, because it gets made in meetings and it is refutable. Saying that a bypass makes rotation and audit meaningless, and therefore that 94% is worth nothing, treats the whole table as if it had only the top three rows. The 94% is worth a great deal: it is the difference between knowing where $180,000 a year of model spend goes and knowing where none of it goes, between charting error rates across every team's traffic and charting nothing, between a quota that protects the provider account from a runaway loop in twenty-one services and one that protects it in zero. A director who argues the strong version will win the meeting and lose the next one, when a team points out that the platform's own dashboards are built on the 94% they were told was worthless.

The accurate claim is narrower and harder to argue with: three specific guarantees are false today, they are the three the compliance and security commitments rest on, and closing them is a matter of one workload rather than a programme.

Finding the last six per cent#

None of this argument works if you cannot tell what is outside. Traffic you have not measured is not counted in the 5.9%; it is counted in the 94.1% by omission, because the denominator came from adding up what you know about. The practical way to get an honest denominator is to look at egress rather than at application logs, since every direct call has to leave your network to reach the vendor, and the network records that departure for you in flow logs — one row per connection, written by the infrastructure rather than by the application whose coverage is the thing in question.

-- VPC flow logs, last 7 days, egress to provider ranges,
-- excluding the gateway's own subnet.
SELECT
  src_workload,
  ANY_VALUE(dest_domain)          AS provider,
  COUNT(*)                        AS connections,
  SUM(bytes_out)                  AS bytes_out
FROM flow_logs
WHERE dest_domain IN ('api.anthropic.com', 'api.openai.com', 'generativelanguage.googleapis.com')
  AND src_subnet NOT IN ('10.42.0.0/20')          -- gateway replicas
  AND ts > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
GROUP BY src_workload
ORDER BY connections DESC;

Running that at the company in question returned three rows rather than one: the enrichment job, a data science notebook host whose users had been calling models interactively for months, and a staging deployment of a service that had been migrated in production but not in staging. The 5.9% figure had been calculated from the enrichment job's own logs, so two of the three were invisible to the audit that produced it. This is the ordinary case. A coverage number derived from the systems you already instrument is a number about your instrumentation.

Egress control is the enforcement that follows the measurement. Denying direct connections to provider domains at the network layer, with the gateway's subnet as the sole exception, converts the policy from a request to a property, and has the useful side effect that a new workload trying to bypass the gateway fails immediately at development time rather than silently in production six months later.

Where the all-or-nothing argument stops#

The reasoning above rests on a specific condition that is easy to lose track of: the uncovered path uses your provider account. That is what makes the credential shared, the traffic in-scope for your audit, and the vendor relationship a single thing you can switch off.

Consider a different non-compliant workload at the same company: a team running its own contract with a different vendor, on an account funded from their own cost centre, with keys they issued and hold. Revoking their access is not something you can do, because you never granted it. Rotating their key contains nothing of yours, because their key does not work against your account. Their traffic was never in your audit trail and its absence does not make your trail incomplete, because your trail's scope is your organisation's usage and theirs is not part of it. Your kill switch never covered them and never claimed to.

That workload may still be a problem. It is a vendor with no data processing agreement, spend nobody has forecast, a model choice nobody has reviewed, and an incident that will land on the platform team's pager anyway when it goes wrong. Those are real governance questions with real answers — procurement, a DPA, a spend review. They are not the same question, and the mistake to avoid is applying an argument whose justification is "one path outside means none holds" to a case where the path was never inside. Chapter 7 turns that distinction into a policy you can actually write down, because a rule that treats these two workloads identically is a rule that produces compliance in the document and shadow deployments in the org.

Coverage is necessary and not sufficient#

Reaching 100% of paths does not deliver the three guarantees on its own; it only removes the thing that makes them impossible. An audit trail is complete only over the fields you actually record, and a gateway that logs token counts but not the model version, or the principal but not the feature, will answer some questions and not others. A kill switch works only if it is exercised — a config flag that has never been flipped in staging is a hypothesis. Rotation is one revocation only if the revocation path is tested and its latency is known, which is the point the previous chapter ended on.

There is also traffic you will never cover and should stop counting as a gap. Your CRM vendor's built-in summarisation feature calls a model on their infrastructure with their key; so does the code assistant your engineers use in their editors. Those are third-party processing relationships governed by contracts, and no amount of gateway adoption brings them inside. Writing them down as out of scope, explicitly, keeps the coverage number meaningful — an in-scope figure of 100% with a named exclusions list is a far more useful artefact than 94% of an undefined total.

With every path on the gateway, the controls that were impossible become merely difficult, and the first one people reach for is the budget. A per-team monthly limit sounds like arithmetic: sum the cost of each request, compare against a number, refuse when it is exceeded. The gateway sees every request, so it should be able to do this exactly. The first month it runs, the number it reports and the number the provider invoices will not match, and the direction of the mismatch is the one that costs money.