Eleven routes, and the spend lives in two of them

The contract analyser costs $0.42 a request. Everyone on the team knows this, because it is the number that comes up whenever anyone opens the traces: the route loads an entire agreement into the context, runs four sequential passes over it, and re-sends the full document on every pass. It is the most obviously wasteful thing in the product, three engineers have independently offered to fix it, and one of them has a design that would halve it.

The route bills $2,268 a month out of $74,114. Halving it saves $1,134, which is 1.5% of the product's model spend, for work estimated at two to three weeks because the four passes carry a shared state machine that would have to be redesigned. In the same product, a chunk deduplication change to the main assistant route — a day of work, no model involved — cuts its input tokens by about 3% and saves $1,153 a month. The two savings are the same money. The efforts differ by an order of magnitude, and the reason is not that one team is cleverer; it is that the two routes carry different fractions of the bill and nobody had looked.

Ranking by the bill, not by the receipt#

Grouping a month of the per-call ledger by route gives the following. The rightmost column is cumulative share, which is the column that ends the argument.

Two routes carry 71.6% of the spend. Four carry 85.6%. The bottom five routes together are $2,508 a month, which is less than the single nightly reporting job, and one of them — the admin explanation endpoint, which produces a plain-English description of a permission change — costs twenty-one dollars a month and has been raised twice in planning as "somewhere we could save".

Concentration this severe is the normal case rather than this product's peculiarity, and the reason is structural. Per-request cost varies across routes by maybe two orders of magnitude, bounded below by the smallest useful prompt and above by the context window, so the spread between the cheapest and most expensive route here is 525×. Request volume varies by whatever the product's usage pattern says, and that is bounded by nothing: a route attached to a keystroke or a page load runs millions of times a month while a route attached to a deliberate human action — uploading a contract, opening an admin panel — runs thousands. The volume spread here is 2,300×, and because total spend is the product of the two, volume wins whenever the spreads are comparable. Any product with one high-frequency surface and several deliberate ones will show this shape.

The per-request column is what makes the ordering counterintuitive. assistant.turn is the cheapest route in the product per call and by far the most expensive in total. contract.analyse is 135× more expensive per request and contributes a twentieth as much. Sorting the table by the $/request column produces almost exactly the reverse ranking of where the money is, and that column is the one that shows up in a trace viewer, in a latency dashboard and in the anecdote an engineer brings to standup. The receipt for one call is visible; the sum over the month is not, unless someone computes it.

The arithmetic that decides where the week goes#

Concentration puts a hard ceiling on what work outside the top routes can achieve, and the ceiling is worth computing before anyone commits a sprint. Deleting routes three through eleven from the product entirely — not optimising them, removing the features — saves $21,034, which is 28.4% of the bill. A mandate to reduce spend by 30% therefore cannot be satisfied without touching assistant.turn or search.summarise, no matter how much effort is spent elsewhere and no matter how wasteful the other nine look.

The same arithmetic run the other way sets the exchange rate between routes. One percent off assistant.turn is $384 a month. One percent off translate.ui is $2.31. A change to the assistant route that is 166× less effective per unit of effort still wins, which is why the natural instinct — start where the inefficiency is most flagrant — is expensive. Flagrancy correlates with per-request cost, and per-request cost is uncorrelated with total spend when volumes span four orders of magnitude, as they do here.

The exchange rate is also the most useful thing to have in your pocket when declining work. "We are not going to optimise the contract analyser" is a position that has to be defended in every planning meeting until someone runs the numbers; "halving the contract analyser is worth $1,134 a month and three weeks, against $1,153 a month and one day on the assistant route" ends the conversation and, more importantly, ends it without anyone having to be wrong. The three engineers who offered to fix the contract route were all correct that it is wasteful. The table is what turns that into a scheduling decision rather than a disagreement about engineering taste.

This is the same reasoning a backend engineer already applies to query optimisation. A query that takes 4 seconds and runs once an hour is a worse target than a query that takes 3 milliseconds and runs forty thousand times a minute, and no amount of the first query being obviously badly written changes the ordering. The profiler exists because human attention lands on the slow-looking thing rather than the total-time thing. A cost ledger grouped by route is the profiler.

Getting the number when nobody tagged the calls#

The grouping above requires that each ledger row know which route produced it, which most systems do not record until someone needs this table. Two mechanisms get you there, and both are worth having.

The provider-side one is a metadata field on the request, which the usage export carries through, so the attribution survives even for calls your own ledger missed:

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    system=[{"type": "text", "text": SYSTEM_PROMPT,
             "cache_control": {"type": "ephemeral"}}],
    messages=messages,
    metadata={"user_id": tenant_hash},
    extra_headers={"x-route": "assistant.turn"},
)

The application-side one is the route column on the ledger from the previous chapter, which is more flexible because you control the taxonomy and can backfill it. The query is unremarkable, and the cumulative column is the part that is easy to leave out and hard to argue without:

SELECT route,
       count(*)                                        AS calls,
       round(sum(cost_usd), 2)                          AS usd,
       round(100 * sum(cost_usd) / sum(sum(cost_usd)) OVER (), 1) AS pct,
       round(100 * sum(sum(cost_usd)) OVER (ORDER BY sum(cost_usd) DESC)
                 / sum(sum(cost_usd)) OVER (), 1)       AS cumulative_pct
FROM   call_ledger
WHERE  ts >= date_trunc('month', now() - interval '1 month')
  AND  ts <  date_trunc('month', now())
GROUP  BY route
ORDER  BY usd DESC;

Choosing what counts as a route takes a little care in both directions. Too coarse and the table hides the answer: labelling everything by service name gives three rows, the largest of which is 60% of spend and tells you nothing about which of its nine code paths to look at. Too fine and it fragments: one route per prompt template produces sixty rows where the top entry is 9%, and the concentration that was there in the data disappears into the taxonomy. The granularity that works is the level at which a lever applies — one route per distinct prompt-plus-model-plus-purpose combination, which is the unit you would change if you decided to shorten output or route down a tier. In this product that produced eleven rows, which is a table a room can read.

The first time this runs, expect a large unattributed bucket — 30% to 50% is typical — from background jobs, scripts, a notebook someone left running, and the evaluation harness. That bucket is a finding rather than a data quality problem to be quietly dropped. In this product it was 11% on the first pass, and two thirds of it turned out to be a nightly re-embedding job that had been re-embedding the entire corpus instead of the changed documents since a refactor eight weeks earlier. That job was $5,400 a month, it belonged in no route, and it had never appeared in any conversation about cost because nobody had a view in which it could appear. Fixing it took one line and it was the largest single saving anyone found that quarter, which is a slightly deflating result for a chapter about ranking levers and an accurate one: the first grouping query frequently finds waste rather than a lever, and the waste should be fixed before the ranking is taken seriously, because $5,400 sitting in an untagged bucket distorts every share in the table.

Where the concentration table misleads#

A monthly snapshot ranks the present and says nothing about the trajectory, and the trajectory can invert the ranking inside a year. onboarding.suggest is $1,760 a month and growing 31% month over month because it was launched six weeks ago and is being rolled out to cohorts. At that rate it is $19,600 a month in nine months, which would make it the third largest route in the product. Optimising it today is still the wrong call, because the design will change three times before the rollout finishes and the work would be redone; noting that its unit cost needs to be right before the last cohort lands is not.

Route is also only one axis of concentration, and often not the sharpest. Grouping the same ledger by tenant shows that 3 of 4,100 accounts generate 22% of assistant.turn traffic, or $8,460 a month, and all three are on a flat-rate plan priced when the feature was a beta. That is a pricing question rather than an engineering one, and no amount of prompt work will resolve it; the cheapest available fix is a fair-use limit in the contract, which takes a conversation rather than a sprint. Grouping by hour shows the nightly report producing a concentrated spike that matters for rate limit headroom and not at all for spend. Grouping by model version, on a product that has been running for two years, occasionally finds a deprecated model still serving 4% of traffic at a price that was competitive eighteen months ago.

The table you build is the one that answers the question you have, and "which route" is the right first question only because the levers in this topic are applied per route. It is worth running the other groupings once anyway, because each takes about a minute against the same ledger and any one of them can turn up something that makes the route ranking moot.

The most consequential thing the table does not contain is quality. Every dollar in it bought an answer, and some of those routes are load-bearing for revenue while others are conveniences. Spend concentration tells you where a percentage point is worth the most; it does not tell you which routes can afford to give one up. That is a property of the lever you apply and the traffic you apply it to, and the levers differ from each other far more than their headline percentages suggest.