A 97% approval rate is a measurement of your operators

An approval rate is the fraction of issued approval requests that an operator approved rather than rejected, and it is the first number anybody computes once a gate has a history. It is read off the stream a durable approval record leaves behind: every request that fired, each with an actor, a latency, an outcome and the card the operator was shown. For the support agent this topic has been following — a B2B analytics vendor's agent, gated on refunds, on replies carrying a commitment, and on subscription cancellations, with a four-person support rota answering the cards — that stream is the only evidence anyone has about whether the gate is doing anything.

Reading the rate as a verdict on the agent fails because the agent is one of two inputs to it. Approval rate is produced jointly by how well the gate is targeted and by how much attention each card receives, and those two inputs are not separable from the rate itself. Two very different systems produce 97%.

Two systems that produce the same percentage#

That table is best read as gaps rather than as levels, and the first thing to notice is which row carries no information. The approval rate is high on both sides by construction: the table enumerates the ways one observed rate can be produced, so the rate is the one quantity guaranteed to be identical in both columns. Everything it teaches lives in the five rows where the worlds disagree, and each of those is a measurement somebody can go and take.

The last row is different in kind, and it is the one that reframes a headline rate, because it is a statement about a direction rather than a level. Suppose the gate's targeting gets better: it fires only on the genuinely borderline cases, so fewer cards arrive, each is harder, and a harder card is one an operator is more likely to send back. The rate goes down. Suppose instead that the queue saturates: more cards arrive than anyone can read, the marginal card gets less attention than the last, and an unread card is approved. The rate goes up. The two mechanisms push it in opposite directions, which makes the sign of its change under rising volume a test that needs neither mechanism quantified. Volume tripled and the approval rate went up, which is what a load effect looks like; a gate whose targeting improved would have produced fewer, harder cards and a lower approval rate.

The belief this chapter exists to dislodge is that a high approval rate is evidence the gate is working, and it is tempting for a good reason: a well-targeted gate on an agent that is usually right genuinely does produce near-unanimous approvals. World A is a real system, and it is the one a team is trying to build. Run the reasoning as a chain and watch where it stops. A gate fires on actions somebody judged worth checking; operators check them; almost all are approved; therefore almost everything the agent proposed was fine. Every link holds inside World A. What the chain never touches is the second input: the approval rate is a quantity that stays high while the dangerous property varies underneath it. Worse than staying constant, it moves the wrong way, because what destroys attention per card is volume, and volume is exactly what drives the rate toward a hundred percent. An operator who reads every card and an operator who clears the queue with the keyboard both produce approvals, and the rate counts them identically.

Four seconds against what the card costs to read#

The time-to-decision distribution is the cheapest of the discriminating measurements and the one that most directly contradicts a high rate. The arithmetic is worth walking rather than asserting, because what follows is an inference from a rate and a length rather than an accusation about anybody's diligence.

A card with a warning line, a payload block, a precedent block and an expiry statement runs to about 90 words with structure. At a generous 300 words per minute of skimming, reading it takes 18 seconds, and that is a floor rather than an estimate of the real cost: 18 seconds buys one pass of the eyes over every word, with nothing checked against anything. Reading it attentively means leaving the card — checking the invoice id against the ticket, noticing that the precedent line says three refunds in 90 days — and each excursion costs more than the words do, which is why attentive reading lands closer to 30. A 4-second median therefore cannot be a read of the card at any speed a person is capable of. It is a recognition response to a familiar shape, which is the same conclusion the architectural mitigations topic reached from a 1.4-second median on deploy confirmations. The shape being recognised there was a deployment banner rather than an approval card.

The direction of travel matters as much as the level. A median of two seconds and falling, on any card worth issuing, is rubber-stamping regardless of what the approval rate says, and the response to it is not to redesign the card first but to go and look at whether rejections correlate with anything — because if they do not, the gate is currently providing no filtering and every hour spent on it is being spent on a control that is off.

What the median hides is the shape, and the shape is more informative than the centre. Healthy decision times are right-skewed with a long tail, because most cards are routine and a few take a minute while somebody opens the billing console. A distribution that is tight around four seconds says every card is receiving the same treatment, which is the signature of a queue being cleared rather than read. Plotting p10, p50 and p90 together, per action class, is enough; the useful alert is on p90 falling rather than on the median, since a p90 of 9 seconds means the hardest decisions are also being made in under ten. A median only moves when half the population moves, while the ninetieth percentile is defined by the slowest tenth — which is where the difficult cards sit if anybody is finding them.

Whether rejections correlate with anything at all#

If operators are discriminating, their rejections should cluster somewhere, because the population of cards is not uniform. Crossing the 2.9% rejection rate against every dimension available is a query anyone can run in an afternoon, and it produces the second piece of evidence.

SELECT action_class,
       width_bucket(amount_cents, 0, 500000, 5)      AS amount_bucket,
       count(*)                                       AS cards,
       avg((state = 'rejected')::int)                 AS reject_rate,
       percentile_cont(0.5) WITHIN GROUP (ORDER BY
         extract(epoch FROM decided_at - created_at)) AS p50_seconds
  FROM approvals
 WHERE decided_at > now() - interval '90 days'
 GROUP BY 1, 2
 ORDER BY 1, 2;

The result on this agent is the finding, and it is a negative one.

Two things about the counts have to be settled before anything is read out of the rates. The first is whether they belong to the window the query asked for. The interval in the WHERE clause is ninety days, a little under thirteen weeks, and at the volume this gate was running at that window can have held at most a bit over twelve thousand cards. The five action-class rows sum to a little over eleven thousand, which sits under that ceiling and close to it — what a window whose earlier weeks ran at a lower volume should produce. The check is a bound rather than a reconciliation: it rules out a table describing some other period, and it does not establish that the five classes are exhaustive.

The second is that the last two rows are a different partition of the same population, not two more slices of the first. Refunds, gated replies and cancellations divide the cards by what the action was; the two time bands divide the same cards by when they arrived, and every morning and evening card was already counted once under its action class. Adding all seven rows counts most of the population twice and still omits the hours nobody sliced. The two partitions share a table because they are read the same way, and a reader who treats the seven rows as one list gets a total that means nothing.

With the counts settled, the rates say something, and what they say is that they are all the same. A rejection rate that is 2.9% everywhere, on a population that ranges from an $8 refund to cancelling an enterprise account, is not a judgement being exercised. The step underneath that reading is short and checkable. If operators were allocating attention by what was at stake, the classes where a wrong action is expensive would collect more rejections than the classes where it is trivial, because that is what allocating attention means. A rate flat across the whole range rules that out. Whatever process generates those rejections is independent of the content of the card, which is what you would expect from misclicks, from the occasional operator who happens to know the customer, and from cards that arrived while somebody was already annoyed. Correlation is the evidence that attention is being allocated; its absence is the evidence that it is not — a measurement read for its variation rather than its level, which is why a team that only ever looks at the aggregate rejection rate can never find it.

The opposite result is equally informative. A rejection rate of 18% on refunds above $1,000 and 0.4% below it says the gate on small refunds is doing nothing and should be removed, which is a conclusion in the same direction as a proposal to stop gating them at all, arrived at with evidence rather than from the headline rate.

Seeded probes and what actually happened afterwards#

The two diagnostics above describe behaviour. Neither says what the gate would catch, and there is a direct way to measure that: inject synthetic cards with payloads that should be rejected, at a known rate, and count how many are. One in 150 cards, generated by perturbing a real payload — a recipient outside the customer's domain, an amount that is ten times the invoice balance, a refund against an already-credited invoice — gives roughly six probes a week at this volume. Over a quarter, 41 probes produced 6 rejections, a catch rate of 15%, and that number is the honest estimate of the protection the gate provides against exactly the class of error it exists for.

What the world imposed and what somebody chose are worth separating here. The injection rate of one in 150 is a design decision, answering two constraints that pull against each other: probes frequent enough to accumulate a countable number of results before the answer goes stale, and rare enough that an operator does not come to expect one. At the roughly 940 cards a week this gate was issuing, one in every 150 works out at about six a week. The catch rate is not a decision at all — six rejections out of forty-one probes is fifteen percent, and that division is the whole of it.

That measurement is the closing identity of this chapter, and what it licenses is worth stating exactly. Approval rate and decision time are observations of behaviour, and behaviour has to be argued from; a probe is an observation of the control, because the correct answer is known before the card is issued. The licence is a direct estimate: at a known injection rate, the fraction of probes rejected estimates the fraction of genuinely bad payloads the gate would stop, in exactly the class of error the gate was built for. Nothing else here gives you that number.

The residual is three things. The estimate is only as good as the probes: one easier to spot than a real bad payload overstates the protection. It is only as good as the sample, and six a week is small, so the rate moves around and should be read across a quarter. And it belongs to the system rather than to the person, which is also the third constraint below.

Three constraints make probes safe to run, and they are conditions on that licence rather than housekeeping. The executor must recognise a probe and drop it, so an approved probe has no effect beyond the log entry; without that the measurement is an experiment run against customers. The team must know that probes exist, even though they do not know which cards are probes, because a hidden test of individuals is a different thing from a measurement of a control. And the results belong to the system rather than to the person: a 15% catch rate is a finding about card design and queue volume, and reporting it per operator turns a diagnostic into a performance review and destroys the honesty of the response.

The last measurement is the only one that says anything about the agent, and it is the one a rate-and-latency slide is missing entirely. Post-action outcomes ask what happened to the actions that were approved: refunds later reversed, replies that produced a complaint or a correction within seven days, cancellations followed by a reinstatement, tickets that were reopened. On this agent, 0.8% of approved refunds were reversed within 30 days against 0.1% of ungated account credits, and the reversals were concentrated on accounts with an open dispute — a precondition the card was not checking. That finding is actionable in a way that no approval rate is.

Those outcomes carry a cost the others do not, and it is a property of the world rather than a choice anybody made. A reversal, a complaint or a reinstatement happens when a customer or a colleague notices, and noticing takes days to weeks, which is why the useful window here is 30 days. No instrumentation shortens it, so the first month of any deployment has no outcome data at all.

The two rows at the bottom of that table are the ones teams have and do not look at. An expiry rate says whether the people named as owners are actually available, and it moves with rota staffing, holidays and incident load rather than with anything about the agent; a jump from 1% to 6% during a week when two of four operators were pulled onto an outage is a capacity finding, and treating it as a discipline problem gets the wrong fix. Edit rate and edit size, from the diff corpus — the stored record of what operators changed when they edited a plan rather than accepting it — point at where the agent's plans are systematically wrong, and they have the useful property of rising when the editing experience improves, which means they can only be compared against themselves within a stable UI version.

Anything computed per operator needs a decision about what it is for before it is computed, because the numbers exist whether or not you look at them. Comparing one person's median decision time against the team's is diagnostic when it identifies someone who has been handed 60% of the queue, and corrosive when it becomes a metric people are measured on, since the cheapest way to raise your apparent diligence is to wait ten seconds before clicking approve. The distinction that holds up in practice is that per-operator numbers are used to rebalance load and to find cards that route to the wrong people, and aggregate numbers are used to judge the control. That split is a design decision rather than a discovery — the same query answers both purposes, so nothing technical stops a per-operator median from becoming a target — and it is worth writing down next to the dashboard.

Where these rates stop answering#

The metrics do not answer everything. They say nothing about the actions that never reach a gate, which is most of what the agent does, and a gate cannot measure the class of error it does not fire on. Every number in this chapter is conditioned on a card having been issued, so the entire ungated population sits outside the sample by construction — which is the reason to read post-action outcomes on ungated actions too, and the reason the 0.1% reversal rate on ungated account credits belongs on the same line as the 0.8% on gated refunds.

A low approval rate is not automatically good either, since a badly targeted gate that fires on nonsense produces rejections that look like vigilance. The rejection-correlation test separates the two: rejections concentrated on an identifiable class are operators finding something, and rejections scattered across a class that should never have been gated are operators dismissing noise.

Probe catch rate is only as meaningful as the probes, and a probe set that has not been redesigned in a year becomes a pattern the team recognises the way they recognise everything else. What ages is the population around the probes rather than the probes themselves, which is why a quarterly design review sits on the metrics table, and why it is the maintenance most likely to be dropped: nothing breaks when it is skipped and the number keeps being produced.

The rates also confound with each other. Expiry rate moves with rota staffing rather than with the agent, so a week of high expiries and a week of high approvals can be one week seen from two angles. The defence is what the table's last column encodes: every rate has a second rate it should be read with, and none is a verdict on its own.

What these rates are good for is deciding where the gates should be next quarter, and the decision they feed is more granular than on or off. A class approved 99.6% of the time with no reversals over 800 actions is asking to leave the queue; a class where operators reject a fifth of the cards, or edit half the plans, is asking for more scrutiny than a card can provide. They are inputs to a placement decision made repeatedly, and a headline approval rate is a poor input precisely because it collapses the two things that decision has to separate.

Turning those observations into a rule requires something that decides, per action rather than per class, which of three fates an action gets.