AI gateways and proxies

Read these first

• Multi-provider abstraction

What you will know

• Route all model traffic through one controlled point

  A bypass path makes credential rotation, audit coverage, and emergency provider blocking incomplete.

• Manage keys centrally rather than per service

  Use a secret manager and short-lived credentials where supported; leaked static keys turn one compromised service into a broader incident.

• Enforce a per-team budget at the gateway

  Accounting must cover retries, streaming, failed requests, and token usage or teams can exceed limits without realizing it.

• Measure gateway latency and failure behavior

  Record added p50 and tail latency, timeout propagation, retry amplification, and behavior when the gateway or provider is unavailable.

• Decide against a gateway with a reason

  For a small or isolated workload, direct provider access may reduce moving parts, but it gives up centralized policy and audit controls.

Chapters

1. 1

   A library cannot hold a key the caller never sees

   A shared abstraction with three vendor implementations already gives you retries and error normalisation, and it will always beat a network hop on latency. What it cannot do is hold a credential the calling service never receives, enforce a budget across services that know nothing about each other, or block a provider without redeploying everything. Keeping both is reasonable, provided you decide which layer owns retries and which owns the key.

   Read chapter
2. 2

   Revoking one service beats rotating a shared secret

   Both designs put a secret in a vault, both are rotatable, both are audited, so the argument turns on what containment costs. One provider key per vendor held at the gateway, plus a gateway token per calling service, makes cutting off a compromised caller a single revocation. Provider keys distributed to services turn the same incident into a rotation coordinated with every holder, at their pace, while the incident is running.

   Read chapter
3. 3

   Rotation, audit and kill switch are all-or-nothing

   An audit finds 94% of model traffic on the gateway and the remainder in a nightly enrichment job holding its own key. Coverage sounds like a percentage you close incrementally, and for three guarantees it is not: rotating a leaked credential, producing a complete audit trail, and blocking a model in an emergency each hold on every path or on none. Where that reasoning stops applying is worth knowing too.

   Read chapter
4. 4

   The meter undercounts what it never saw finish

   The gateway reports $2,900 against a $4,000 budget while the invoice for the key only it holds reads $6,100, and both systems are working. The gap is usage whose ending the gateway missed: streams abandoned before the final chunk, internally retried calls counted once, generations that completed upstream and failed on the way back. Enforcing rather than reporting means committing to an estimate at admission, and offering something short of a hard cap.

   Read chapter
5. 5

   Two retry layers multiply, they do not add

   A caller that already retries twice on 5xx sitting behind a gateway that also retries twice produces nine provider attempts per request, so a 30-second brownout arrives as nine times normal load and holds the provider past the point it would have recovered. Measure what the hop itself adds at p50 and in the tail, check how the caller's timeout propagates through it, and settle in one place who is allowed to retry.

   Read chapter
6. 6

   A stateless proxy is still a total outage for nine teams

   Three replicas behind a load balancer took every team down for 40 seconds during a routine deploy, and two of those teams had assumed a gateway problem would show up as degraded quality rather than as no feature at all. A second cluster answers that deploy and not the class, since bad config and bad builds reach both. The review item is a rehearsed per-caller answer: who queues, who fails fast, who sheds load.

   Read chapter
7. 7

   Not yet is defensible if you leave the door open

   A two-person team running one provider, one prompt and 3,000 requests a day in a company where nothing else uses models should not take the gateway as their single shared dependency. What you owe them are the conditions that make later adoption configuration rather than migration: the key in the company vault, usage records in the platform schema. The same reasoning separates a bypassing batch job from a team funding its own provider account.

   Read chapter