Dashboards that disagree cannot rank anything

Model spend has grown 6% a month for three quarters, which compounds to 69% higher than it was in September, and at $74,100 a month it is now the second largest line in the infrastructure budget behind managed Postgres. The CFO has asked for 30%. Six product teams ship independently against the same two providers, and two of those teams have built their own cost dashboards. The dashboards disagree with each other by 18% on the same route in the same month, and both disagree with the invoice.

The first meeting about the 30% will therefore not be about the 30%. It will be about whose number is right, it will run for an hour, and it will end with someone volunteering to look into it. Every subsequent meeting will do the same until the numbers reconcile, because a ranking assembled from figures that half the room disputes gets argued rather than executed, and the arguing is cheap for everyone except the person accountable for the target.

Why two honest dashboards disagree#

None of the discrepancies are fraud or incompetence. They are the specific ways a per-call cost estimate diverges from a provider bill, and it is worth knowing them by name because each one has a different fix.

Client-side token estimation is the most common. A dashboard that counts tokens with a four-characters-per-token heuristic instead of reading the provider's usage fields will be wrong by whatever the corpus makes it: on this organisation's traffic, heavy with identifiers and code fragments, the heuristic ran 9% low. Cache accounting produces the largest single error here — one dashboard counted cache reads at the full input price, which overstated the assistant route's input spend by 31%, while neither dashboard modelled the write premium, understating it by about 4%. Streamed responses that the client abandons often never deliver a final usage block, so 2.4% of calls were absent from one dashboard and present on the invoice. Retries performed inside the provider SDK, below the layer the dashboards instrument, added another 3%.

Then there are the boring ones that are just as fatal to agreement. The two dashboards use different month boundaries, one in UTC and one in US/Pacific, which moves several thousand dollars between months. One hard-coded prices from a launch announcement and has been stale since a price change in February. And committed-use discounts, where a contracted level of annual spend buys a lower rate, along with promotional credits, are applied at the invoice level rather than per call, so the sum of per-call list prices is legitimately 8% above what the company pays, which makes the invoice look wrong to anyone comparing it against a correct per-call ledger.

Diagram

The properties that make this arrangement settle arguments are that there is one ledger rather than one per team, that it is derived from the billing export rather than from application-side estimates, that prices live in a versioned table with effective dates so a historical month can be recomputed correctly, and that discounts are a separate modelled line so per-call list-price sums stay comparable across teams while the total still ties out. The tolerance is stated up front — 1.5% here — so a 0.8% gap is not an emergency and a 4% gap has a ticket and an owner rather than a debate. The unattributed bucket from chapter 3 gets an owner too, because a bucket nobody owns grows.

Getting this in place took two engineers three weeks. It produced no savings whatsoever, and it is the prerequisite for every dollar that follows.

The organisational half of the fix is shorter to describe and harder to do: the two teams that built dashboards stop maintaining them, and one of the two owns the shared ledger. Leaving both running "for comparison" preserves exactly the situation that made the first meeting useless, because a team whose number disagrees with the ledger will keep quoting its own in the meeting where its budget is discussed. The migration is easier than it sounds when the ledger is demonstrably closer to the invoice — the argument is settled by the reconciliation report rather than by seniority, which is the main reason to publish the tolerance at all.

A uniform target ignores where the money is#

The obvious way to distribute a 30% mandate across six teams is to give each team 30%. It is fair-looking, it respects autonomy, and the arithmetic appears to work: six 30% cuts sum to a 30% cut.

The uniform column asks the Docs team to find $1,290 a month, which is $15,480 a year, for work that will realistically consume a quarter of an engineer's attention across a quarter — a fully loaded cost, meaning salary plus the overheads an employer pays on top of it, comfortably above the saving. It asks the same of Support and Platform. Meanwhile it asks the Assistant team for 30% when the levers available on that route, from the preceding chapters, could plausibly deliver 45% to 50%, so the team with the most room is given the least demanding number and will hit it in six weeks with the two easiest changes.

The concentrated plan reaches the same total by asking two teams for 40% and the other four for nothing beyond adopting a default-tier policy for new routes, which costs them no project time. Four teams keep shipping product, two teams do focused work with a real budget behind it, and the aggregate lands slightly ahead.

The second problem with the uniform target is not arithmetic. Six teams each finding 30% will each make a quality trade, privately, with whatever evaluation they happen to have, and nobody will hold the aggregate. One team ships a cheaper tier that costs 0.05 points on its slices; another caps output and breaks a workflow the way chapter 5 described; a third quietly narrows retrieval from eight chunks to four. Individually each is defensible and none is written down in comparable form. What arrives at the end of the quarter is a 30% reduction and an unknown, uncomparable, unattributable quality position across the product — and the person who has to answer for that in front of a customer is not any of the six.

"Every route uses the cheapest model that passes its evaluation"#

A team proposes this as company policy, and it has genuine merit. It forces every route to have an evaluation, which most do not; the Docs team's summariser has been running on the flagship model since launch because that was the newest model on the day it was written, and nobody has ever tested anything else. It converts model selection from an inherited default into a decision with evidence behind it, and it gives new routes a required artefact at launch rather than a retrofit two years later.

The objections are about what the sentence delegates. "Passes its evaluation" is defined by whoever writes the evaluation, which is the team with an incentive for the bar to be reachable, so the policy hands the standard to the party it is meant to constrain. Evaluations written once are rarely re-run, so the policy binds at launch and then decays while prompts, traffic and model versions all drift underneath it — and a route that passed in March on a model that has since been silently updated is not a route anyone has evidence about. It optimises every route locally, which is precisely the mistake chapter 3 warned about, so four teams spend a quarter chasing $1,200 in aggregate. It is also silent on latency, reliability and rate-limit headroom, all of which a cheaper tier changes, sometimes favourably and sometimes not. And "cheapest that passes" leaves no margin by construction, so any subsequent prompt edit or traffic shift pushes the route below its own bar, and moving back up means asking for a cost increase, which is a conversation people avoid.

The version worth adopting keeps the forcing function and drops the absolutism: routes above a spend threshold — $2,000 a month covers four of eleven here and 89% of spend — must declare a model, an evaluation set sampled from their own production traffic, and a margin above the passing bar, with the evaluation re-run quarterly and on any model version change. Below the threshold, a default-tier policy for new routes with no project work required.

Keeping it, and what to tell the CFO it cost#

A 30% reduction decays unless the accounting outlives the project that produced it. In this organisation, prompt growth alone added roughly 800 tokens to the assistant route's system block over five months of ordinary feature work, which at that route's volume is about 8% of its spend arriving with nobody's approval. The mechanisms that hold the line are unglamorous: spend per request per route tracked as a metric with an alert on regression, rather than total spend, which grows with success and therefore always looks bad; a CI check that fails a pull request raising a route's tokens per request by more than 10% without an explicit note; the monthly reconciliation with its named owner; a default tier for new routes that requires justification to exceed; and the evaluation sets kept alive, since a quality baseline nobody maintains cannot be compared against next year.

What to report as the cost has four parts. The quality position, stated per workflow in product language — multi-document comparison degraded materially and one workflow's regression was accepted with a named owner — rather than as rubric points. The engineering: about 2.1 engineer-quarters to build, including the three weeks of reconciliation that produced no savings, plus roughly 0.15 of an engineer ongoing to maintain evaluations and the ledger. The capability foreclosed: the nightly report can no longer be re-run interactively, which was a real feature that real users used.

And the fourth part, which is the one that changes what gets asked for next. At 6% monthly growth, a 30% reduction buys back about six months. It is a level shift rather than a change in slope, and the slope is set by product decisions about which surfaces get a model call in them, not by the levers in this topic. The portfolio you now have — a reconciled ledger, a concentration ranking, per-lever savings with their bases named, and a quality record per slice — is what lets that conversation happen with numbers on the table instead of percentages from a vendor post.