Server-side rollup of log lines

A rollup is a tool result that carries computed aggregates in place of the records they were computed from: counts, percentiles, group-bys, and a small, deliberately chosen sample of the underlying rows. It is the response shape for every question whose answer is not the rows — why did latency move, how much was refunded, which services were involved — which is most of what an agent is actually asked about a large table.

The alternative is not sending the rows, because the rows do not fit. A structured log line in a checkout service is about 214 characters — timestamp, level, trace id, span id, message, a few labelled fields — and measured with the token-counting method from chapter 2 it comes to 96 tokens, because a log line is mostly identifiers and machine-formatted times. An hour of one service at 41,300 lines is therefore about 3.96 million tokens, roughly twenty times a 200,000-token window and about $11.90 of input at $3 per million if any window could hold it. Capping that at the 500 most recent lines makes the call succeed and is not a smaller version of the same answer; it is a different and much worse one.

Each of those figures is one multiplication, and doing them rather than reading them is what makes the scale believable. 41,300 lines at 96 tokens a line is a little under four million, which is the 3.96 million. Dividing that by a 200,000-token window gives just under twenty, so the hour would have to be cut into twenty full windows with nothing else in any of them — no instructions, no tool definitions, no question. Multiplying by $3 per million gives the $11.90, and the conditional attached to it is load-bearing rather than decorative: that is what the hour would cost if a window could hold it, and none can, so the price is a measure of size wearing the units of money. Keeping all three is worth the room because they fail as arguments in different places. The token count says the payload is large, the window ratio says it is impossible, and the price says it would be a poor idea even if it were not.

The cap is tempting because it looks like a resolution setting: fewer lines, same picture, slightly blurrier. It is not, and the reason is that a cap selects on one axis while the question selects on another. Taking the most recent 500 lines orders the corpus by time and keeps the end of it. A latency incident is a question about the minutes in which something anomalous happened, and recency and anomaly are unrelated properties of a log line — nothing makes the interesting minute the last one. So the cap does not lower the resolution of the answer. It substitutes a different sample, chosen by a criterion the question never mentioned, and when the anomaly is not at the end of the window the sample contains none of it. A blurry picture of the right thing would be useful. This is a sharp picture of something else.

What the agent needs from an hour of log lines#

A site reliability engineer — an SRE, the person on the rota who is paged when a service misbehaves — handed a latency incident does not read the hour either. They run three queries in the log store: a count by level, a latency histogram bucketed by minute, and a group-by on error class. Then they look at which minute the tail moved in, pull the distinct exceptions in that minute, and read those in full. The whole procedure touches maybe forty raw lines. Everything else is consumed as counts, and the counts are computed by the store, which is built to do exactly that over an hour of data in a few hundred milliseconds.

That procedure is the response shape. The tool should return what the queries return rather than the corpus the queries ran over:

{
  "basis": {
    "service": "checkout",
    "window": ["2026-03-17T14:00:00Z", "2026-03-17T15:00:00Z"],
    "filter": "env=prod",
    "lines_matched": 41300,
    "lines_returned_verbatim": 3
  },
  "by_level": { "INFO": 39112, "WARN": 2129, "ERROR": 59 },
  "latency_ms_by_minute": [
    { "minute": "14:03", "p50": 88, "p95": 210, "p99": 244, "n": 702 },
    { "minute": "14:05", "p50": 91, "p95": 2840, "p99": 3940, "n": 688 },
    { "minute": "14:16", "p50": 87, "p95": 226, "p99": 251, "n": 715 }
  ],
  "error_classes": [
    { "class": "PoolTimeoutError", "count": 47, "first": "14:05:11Z", "last": "14:15:52Z" },
    { "class": "UpstreamRefused", "count": 12, "first": "14:05:09Z", "last": "14:05:44Z" }
  ],
  "distinct_traces": [
    {
      "count": 47,
      "example_trace_id": "b41f9c2a7d",
      "text": "PoolTimeoutError: connection pool 'orders-ro' exhausted (size=8, in_use=8, waiters=61)\n  at Pool.acquire (pool.js:212)\n  at OrderRepo.findByCustomer (order_repo.js:88)\n  at handleCheckout (checkout.js:141)"
    }
  ],
  "drill_down": {
    "tool": "get_requests",
    "args": ["minute", "min_duration_ms", "limit"],
    "note": "returns raw request records behind any figure above"
  }
}

That payload is 1,150 tokens against 3.96 million, and it contains the answer. The p99 column shows the tail opening at 14:05 and closing at 14:16 while p50 barely moves, which is the signature of contention rather than a general slowdown; the error classes name the contention; the stack trace names the pool, its size and the sixty-one waiters queued behind it. An agent given this writes a correct explanation in one turn.

Why a flat median beside a moving tail reads as contention rather than as slowness is the step that paragraph asserts, and it is short enough to walk. A connection pool holds a fixed number of connections; this one holds eight. A request that finds a free connection is served at the service's ordinary speed, and nothing about the pool being busy makes that request slower. A request that finds none waits until somebody returns one, so its total time is its own work plus however long it queued. The population therefore splits in two rather than shifting as a whole. While most requests still find a connection, the middle of the distribution sits where it always sat, which is the p50 hovering around 88 and 91; only the requests that queued move, and moving them puts them in the tail. A general slowdown looks nothing like that, because if every request got slower the median would move first and most visibly — there are more requests near the median than anywhere else, which is what makes it the median. So a flat p50 beside a p99 that has climbed more than tenfold is the shape of a queue, and waiters=61 is that queue counted.

Two details in that payload are doing more work than their size suggests. The stack traces are reproduced verbatim, not normalized into a class name and a count, because the line numbers and the interpolated values — size=8, in_use=8, waiters=61 — are the evidence, and a hash of the trace is a key for a system that has the traces rather than for a model that does not. The count next to each trace is what separates the 47-occurrence pool timeout from a single stray ValueError in a background job, which is a distinction the model cannot make from three traces presented as equals. Three distinct traces at roughly 90 tokens each is 270 tokens for the part of the payload with the highest answer density in it.

The basis block also makes the rest of the payload checkable, which is a property worth using rather than merely providing. The three entries in by_level add to 41,300, which is the same number lines_matched reports, so the counts account for every line the filter matched and nothing was quietly dropped between the query and the serialiser. The two entries in error_classes add to the 59 that by_level reports as ERROR, so the grouping covers the errors rather than sampling them. Neither identity is guaranteed by anything in the format. They are properties of a correctly written aggregation, and the difference matters: a payload where they hold can be reasoned from, while a payload where they quietly do not is an authoritative-looking set of numbers about a population nobody can name.

Where the aggregation runs is a separate question from whether it runs, and it matters less than teams expect. Pushing it into the log store is the efficient version — Loki, ClickHouse and their equivalents compute percentiles and group-bys over an hour of a single service in 200 to 600 ms without moving the rows anywhere. Streaming the 41,300 lines into the tool process and aggregating them in Python is slower and heavier, on the order of 1.5 seconds and 60 MB of transient memory in this service, and it is still an entirely acceptable implementation, because both versions keep the rows on your side of the boundary. The arrangement that fails is the one where the rows cross into the context window and the model does the counting, which is the slowest, most expensive and least accurate of the three by a wide margin.

The dotted edge is the part teams leave out. Aggregation on its own is a one-way door: the model sees a p99 of 3,940 ms at 14:05 and has no way to ask which requests produced it, so it either accepts the number as the end of the analysis or asks for the whole hour again. get_requests(minute="14:05", min_duration_ms=3000, limit=20) returns the twelve requests in that minute that exceeded three seconds — twelve records, roughly 900 tokens — and those twelve turn out to share a customer id, which is the fact that finishes the incident.

Four thousand transactions to answer one arithmetic question#

A finance agent has a version of the same problem with none of the log-shaped detail. Asked how much the company refunded in March, it calls list_transactions(type="refund", month="2026-03"), which matches about 4,000 rows. At the per-row costs from chapter 2 those rows are roughly 1.2 million tokens, so no single page holds them and the agent must either page eighty times or receive a capped sample.

Both of those numbers come from the one per-row cost, and the second is the one that decides the design. Four thousand rows at the just-under-three-hundred tokens a row that chapter 2 measured is a little under 1.2 million, which is several windows' worth. At the fifty-row page size this tool uses, four thousand rows is eighty pages. Pages are not free the way rows are: eighty pages is eighty model round trips, and the token cost of the traversal is not 1.2 million but rather more than that, because every page joins the conversation and is re-read as input on every turn that follows it. The row count sets the size of the problem; the page count sets what it costs to work through.

Paging is the version that looks rigorous and is the most expensive way to be wrong. Eighty round trips at three to five seconds each is four to seven minutes of wall clock, the input cost compounds because each page joins the conversation and is re-sent on every subsequent turn, and at the end a language model is adding four thousand decimal amounts across eighty separate contexts. Models are unreliable at long-chain arithmetic in a way that is uncorrelated with how confident the answer sounds, so the failure mode is a total that is plausible and off by one refund's worth. The database computes the same total exactly, in one query, in a few milliseconds.

{
  "basis": {
    "filter": "type=refund AND status=settled AND month=2026-03",
    "rows_aggregated": 4127,
    "currency": "EUR"
  },
  "total_refunded": 318442.15,
  "mean": 77.16,
  "largest": [
    { "id": "txn_8f21", "amount": 4200.00, "date": "2026-03-08", "customer": "Northwind Foods", "reason": "duplicate charge" },
    { "id": "txn_a09c", "amount": 3150.00, "date": "2026-03-22", "customer": "Halcyon Retail", "reason": "order cancelled" },
    { "id": "txn_5b77", "amount": 2980.00, "date": "2026-03-22", "customer": "Halcyon Retail", "reason": "order cancelled" }
  ]
}

The total answers the question. The row count lets the model say what the total covers, and lets a human notice that 4,127 is a third higher than February. The largest few are there because they are what the next question is about — a €318,442 total is an abstraction, and three named refunds turn it into something the model can discuss, sanity-check against the mean, and follow up on. Two of the three landing on the same customer on the same day is the kind of pattern that only exists in the rows and survives into a well-chosen sample of them.

The mean is in there for a reason easy to overlook, which is that it is the one field a reader can check the others against. Dividing the total by the row count reproduces it: 318,442.15 over 4,127 refunds is 77.16. A payload in which those three numbers fail to reconcile is a payload whose figures were computed over different populations, and the failure is visible to anyone holding the result, without access to a single row. The mean also gives the largest few somewhere to sit. The biggest refund in that list, set against a mean of €77.16, is not a large refund so much as a refund of a different kind — some fifty times the typical one — and the model can say so because it was handed both numbers rather than one.

A rollup is lossy at a boundary you chose#

A capped result declares itself a sample. A rollup is a sample too, in a stronger sense: the raw rows are gone entirely, and what remains is a projection of them chosen by whoever wrote the aggregation. It owes the same four-part contract, adapted to what it did.

The first row of that table is the same multiplication run once more: 500 lines at 96 tokens each is 48,000, which is what a capped page costs. Set that against the 1,150 the rollup costs and the rollup is under a fortieth of the price, and it is also the one that answers the question. Cheaper and more useful is an unusual pairing and worth stating plainly rather than letting it pass. It happens because the two payloads are not the same information at two resolutions. One is a slice of the rows chosen by time; the other is a set of answers computed from all of them.

The basis block is worth its 60 tokens for the same reason matched and returned were worth theirs. A rollup computed over env=prod and one computed over all environments produce identically shaped payloads with different numbers, and a model that is not told which it received cannot tell that the 59 errors it is looking at exclude the staging noise or include it. Aggregates are more dangerous than raw rows in exactly this respect, because a wrong raw row is visibly one row and a wrong aggregate is an authoritative number with no visible provenance.

The most common way a rollup goes wrong quietly is a top-N cut applied to the wrong dimension. Returning the ten most frequent error classes is a sensible default and it systematically hides the singleton, which in incident work is often the one that matters: a pool timeout firing 47 times is a symptom, and the single ConfigLoadError at 14:05:02 that emptied the pool is the cause, ranked eleventh by count and dropped. The cheap correction is to rank on more than frequency — include every class whose first occurrence falls inside the window even if it occurred once, and every class not seen in the preceding hour. That is a handful of extra rows, and it is the difference between an aggregate that describes the noise and one that describes the change.

The queries you did not think to write#

What a rollup can answer is fixed at the moment you write the aggregation, which is the honest limit of the technique. Counts by level, percentiles by minute and grouped error classes answer the questions an SRE asks about a latency spike, and they answer nothing about whether a particular customer id appears in the hour, whether anyone logged the word deadlock, or whether the WARN volume contains a message nobody has seen before. Those are real incident questions, and the tool as designed returns a payload in which the answer has already been discarded.

Some of that is recoverable by adding dimensions — a top_messages list, a distinct-count on customer id — and each addition costs tokens on every call, including the thousands where nobody needed it. The rest is not recoverable in advance, because it depends on a question the aggregation would have had to anticipate. A one-line FATAL: schema version mismatch, refusing writes sits in 41,300 lines as a single ERROR in a count of 59, and no percentile or group-by surfaces it unless someone thought to group on it.

Computing the answer where the rows already are is the difference between a 1,150-token result that explains an incident and a 48,000-token result that misses it, and the arithmetic is the same whether the rows are log lines or refunds. What makes it a designed result rather than a convenient one is the basis block that states the window, the filter and the number of rows it stands for, and the drill-down call that reaches the records behind any figure — because an agent staring at a p99 with no way to ask for the twelve requests that produced it has been handed a conclusion instead of evidence.

That is the case where a cheap model reading the payload starts to look attractive. It does not need the aggregation specified in advance, because it reads the lines and decides what mattered — which is a genuinely different capability from anything in this chapter, and it comes with a cost that is easy to miss precisely because it never shows up as an error.