Lead · 6 chapters
Metrics and SLOs
TTFT, p50/p95/p99, tokens per second, error rate, cost per request — and an SLO on quality, the part with no equivalent in your existing practice
Quality metrics are noisier and easier to game than latency metrics, so a service can meet every technical SLO while users receive worse answers.
Read these first
What you will know
Define SLOs for TTFT and end-to-end latency
Set targets over a stated time window and population, because streaming can hide slow completion behind an acceptable first token.
Define an SLO on quality and say how it is measured
Specify the evaluator, dataset, threshold, and sampling method; judge drift and an unrepresentative sample can make a healthy system look degraded or vice versa.
Instrument the quality SLO with model, prompt, and traffic-slice metadata
Without versioned dimensions, an aggregate score cannot reveal that one model, tenant, language, or request class is failing.
Alert on a quality regression rather than only on errors
Use a minimum sample size and a sustained-change rule so noisy evaluator scores do not page operators for random variation.
Report against the SLOs weekly without manual work
Automated reports should expose budget burn, missing data, and metric changes alongside cost and latency rather than presenting a single blended score.
Chapters
- 1
A first token on time says nothing about the last one
A p95 TTFT objective under 500ms can drift from 310ms to 380ms across a quarter while the same 600-token answer stretches from four seconds to eleven. Streaming moved the user's wait into the decode phase, where no objective was watching. Add time-to-last-token and output tokens per second, accept that answer length legitimately affects the total, and name the window and the population or the figure is an average over whatever arrived.
- 2
An evaluator, a dataset and a sample, or it is not an objective
"The weekly average LLM-judge score is at least 4.2 out of 5" names neither population nor sample, so it moves with the request mix and nothing separates that from a real regression. A usable objective states which requests are eligible, how they are drawn, which evaluator and evaluator version scores them, against what reference data, and what threshold over what window. Three flat months is usually evidence the metric cannot move.
- 3
When the score moves and the system did not
The quality objective sat at 96% for weeks and reads 91% this morning, with no deploy, no prompt change and green provider status, while low-scoring traces show sensible answers to sensible questions. Interrogate the measurement before the system: the judge's own version and prompt template, the sampling filter that selects requests, and whether today's volume of scored samples matches yesterday's. An evaluator is a dependency that changes underneath you.
- 4
An aggregate cannot name the tenant it is hiding
Two months at 96% while one enterprise customer submitting German documents received unusable answers for six weeks, with evaluator, dataset and sampling all working as designed. What was missing is the metadata that makes a score interrogable: model version, prompt version, tenant, language and request class attached to every scored sample. A tenant worth two percent of traffic cannot move the mean however precisely that mean is estimated.
- 5
Fifty samples an hour crosses any threshold on its own
Scoring fifty requests an hour and paging whenever the hourly mean dips below target yields four pages a week, every one closed as recovered without action. That is ordinary variation crossing a line, not a signal, and raising the sampling rate buys precision at the cost of putting a judge model in front of production traffic. An alert needs a minimum sample size before it may fire and a sustained change across consecutive windows.
- 6
Error-budget burn beside the share of traffic measured
Leadership wants one blended AI health score, but three of the four objectives are computed from complete data and the fourth is a sample estimate that was 60% missing the week the evaluator silently timed out. Report each objective with its budget burn and the fraction of eligible traffic actually scored, because a composite cannot express missing data. With eight teams shipping prompts independently, burn has to be attributed by prompt version.
External resources
- NVIDIA NIM LLM Benchmarking: Metricsdocs.nvidia.com
Precise definitions of TTFT, ITL, TPS so your dashboards mean something.
- Monitor LLM Inference in Production with Prometheus and Grafanaglukhov.org
Concrete exporter plus dashboard setup for latency, tokens and errors.
- LLM inference latency: TTFT, tokens per second, and what to measureclickhouse.com
Builds intuition for which percentile matters for which user experience.
- The LLM Inference Trilemma: Throughput, Latency, Costdigitalocean.com
Explains tradeoffs among throughput, latency, and cost for inference performance metrics and SLOs.
- LLM Cost Optimization: 5 Levers That Cut API Spend 70-85%morphllm.com
Explains levers for cutting LLM API costs, with metrics for tokens, latency, and usage.
- Lessons from the Trenches: Building LLM Evals That Work IRL: Aparna DhinkaranAI Engineer
How teams put a measurable target on quality, not just latency.
- The maturity phases of running evals — Phil Hetzel, BraintrustAI Engineer
Shows how a quality SLO evolves as an AI product matures.