Self-hosting a pipeline nobody queries migrates the waste

The finance report that started it showed £13,920 for a single month of LLM observability. The purchase two years earlier had been justified at £42,000 a year on a forecast of 55 million traces a month, and the current run rate — the latest month's invoice multiplied out to a year, which is how finance talks about a bill that is still climbing — was £167,000 on 240 million. Four times the forecast, on a line item nobody had looked at since the contract was signed.

Within a week a staff engineer had a proposal in a document: ClickHouse on three nodes — the column-oriented analytics database most self-hosted tracing stacks are built on, chosen because scanning one attribute across billions of spans is what it is good at — with object storage for cold data, an open-source trace UI in front, roughly £2,900 a month of infrastructure, eight engineer-weeks to build. The headline was that self-hosting would save about £130,000 a year, and it was well researched, with a schema, a retention plan and a benchmark. The platform lead who had bought the SaaS in the first place read it as a verdict on his judgement, and the first meeting about it produced no decision at all.

What neither document contained was any statement of what the 240 million traces were for. Somebody eventually pulled the vendor's query audit log, which covers who ran what against which service, and joined it against ingest volume by service.

The three queries against doc-batch-reindex were all run by the engineer who instrumented it, all in the same week in 2024, and none since. That pipeline runs nightly, emits a span per chunk embedded, and has been the single largest contributor to the bill for eighteen months without anyone ever reading its output.

Self-hosting stores the same junk more cheaply#

The proposal's arithmetic is sound as far as it goes, and doing it properly with the line items from chapter 1 makes the shape clear. Infrastructure at current volume is £34,800 a year. The build is eight weeks, £80,000. Ongoing ownership — retention and compaction, meaning the scheduled work that merges the many small files each ingest batch writes into fewer large ones and deletes what has aged out, plus schema migrations when span attributes change, upgrades, and an on-call rota for a system that has the specific charm of being the thing you use to debug outages while being capable of being the outage — is about 4.5 engineer-weeks a year, £45,000. Steady state is therefore £79,800 a year against £167,000, which is a genuine saving, and over three years £319,400 against roughly £504,000.

Then look at what one afternoon of sampling configuration does to the same comparison. The mechanism is tail sampling, which holds a trace until all of its spans have arrived and only then decides whether to keep it, so the decision can be made on what actually happened during the request rather than on a coin flip at its first span.

# otel-collector: keep everything worth reading, drop the rest at the edge
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: always-keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: always-keep-slow
        type: latency
        latency: {threshold_ms: 2000}
      - name: batch-reindex-sample
        type: and
        and:
          and_sub_policy:
            - name: is-batch
              type: string_attribute
              string_attribute: {key: service.name, values: [doc-batch-reindex]}
            - name: one-percent
              type: probabilistic
              probabilistic: {sampling_percentage: 1}
      - name: everything-else
        type: always_sample

Errors and slow spans from the batch pipeline are still captured in full, which is the only use anyone has ever had for them, and the routine successes are sampled at 1%. Ingest falls from 240 million traces a month to about 99.6 million, and the SaaS bill falls to roughly £69,300 a year at the same unit price. The self-hosting case shrinks with it — smaller cluster, £16,800 of infrastructure — but the build and the ownership do not, so steady state lands at about £61,800 against £69,300, and over three years £265,400 against £222,900 including the week and a half to implement the sampling.

The ranking flips because the volume was never a fact about the business; it was an unmanaged output of one nightly job. The economics did not invert. The bill grew because nothing in the system owned ingest volume, and any pricing model that scales with volume needs someone who is accountable for that volume in the same way someone is accountable for cloud spend. Migrating the workload to your own hardware without fixing that gives you a lower unit price on data nobody reads, plus a compaction job that pages at four in the morning for a table nobody queries. The unit price falls and the total cost of attention rises.

It is worth knowing how the 143 million traces came to exist, because nobody decided on them. The reindex pipeline was instrumented by adding the vendor's auto-instrumentation package to a worker that already emitted a span per unit of work, and its unit of work is a chunk. When the corpus grew from 900,000 chunks to 4.2 million and the reindex cadence went from weekly to nightly, trace volume grew by a factor of twenty without a single line of code changing and without anyone approving anything. That is the general shape of usage-priced infrastructure going wrong: the multiplication happens between two systems that are each behaving correctly, and the invoice is the only place the product of the two appears.

The control that prevents a recurrence is a per-service ingest budget with an alert, in exactly the form teams already use for cloud spend. Each service declares an expected traces-per-month figure at the point it is instrumented, the collector exports its own counters, and exceeding the declared figure by more than 50% opens a ticket against the owning team rather than silently costing money. Setting the initial numbers takes an afternoon across a dozen services and they do not need to be accurate, only present, because the value is in something existing to be violated.

Helicone's Build vs. Buy LLM Observability is worth reading here for its accounting of what self-hosting logging and tracing actually costs at volume, because it is unusually honest about the parts that do not appear in an infrastructure estimate.

Running the review so it produces a decision#

The meeting failed the first time because the agenda contained one question that was really two. "Should we self-host observability?" merges a bug — ingest volume is 2.4 times what anyone would choose, and nobody noticed — with a decision, which is whether this company should own its observability layer. Merging them forces the person who made the original purchase to defend both at once, and one of them is genuinely indefensible, so the whole conversation becomes about him. Split into two agenda items, the first has an obvious answer that costs nobody anything to agree with, and the second becomes a normal architectural discussion.

Four other things make the difference between a review and an argument. The first is that the review runs against the original record rather than against the person: chapter 5's format exists so that the thing under examination is an assumption with a number on it. Here the record said 55 million traces a month, and A1 is broken by a factor of 4.4. That is a checkable statement about the world, and disagreeing with it is not disagreeing with a colleague.

The second is that the person who made the original decision presents its assumptions. This inverts the social position entirely, because he is the only person who knows why the number was 55 million and what the alternatives were in 2024, which makes him the expert witness rather than the defendant. Reviews chaired by the incumbent decider go materially better than reviews sprung on them, and the cost is nothing.

The third is agreeing the decision criteria before anyone presents an option: the horizon, the line items that go in both columns, and the estimate multiplier from chapter 1. Once the columns are agreed, the numbers do the arguing. Doing this after the proposal is on the table means every criterion looks like a move for or against it.

The fourth is a measurement week before the decision rather than after it. The query-log join above took an afternoon and it changed the answer; the sampling change is reversible with one config edit, so shipping it first and re-running the cost comparison a month later costs almost nothing and removes the largest uncertainty from both columns.

Underneath all four is a norm worth defending explicitly, which is that a forecast being wrong two years later is the expected outcome of forecasting, not a performance problem. If reopening a decision costs the original decider status, then the rational move for everyone is to never write a trigger and never volunteer a review, and the organisation loses the mechanism entirely to protect one person's afternoon.

What goes in the record, beyond the choice#

The output of this review is a new numbered record superseding the old one, and the option it picks is the least interesting thing in it. Four other things belong there.

The revised assumptions, with the volume now stated as a controlled quantity rather than a forecast: post-sampling ingest of 100 million traces a month, with a named owner for ingest volume and a per-service budget. The triggers that were missing, wired to a job the way chapter 5 described — annual run-rate above £110,000, monthly ingest above 130 million, or any service exceeding 20% of ingest with fewer than 50 queries in the trailing quarter. The counterfactual, meaning the self-hosting numbers as they were actually computed, so that when someone proposes it again next year the analysis is on the shelf rather than repeated from scratch. And the lesson, written plainly: the original record carried a volume assumption with no owner and no alarm, and a per-service volume-versus-query report now runs monthly for every usage-priced vendor, not just this one.

That last item is the part that compounds, because the same failure was available on the gateway contract and on the vector store, and it was only observability that happened to get expensive enough to notice.

None of this makes self-hosting observability wrong in general. There is a real crossover, and it sits roughly where a SaaS bill on traffic you genuinely read passes six figures a year and you already have a platform team operating ClickHouse or similar for other reasons, at which point the marginal cost of one more workload is small and the saving is real. The mistake in this case was not the destination but the order of operations: nobody had established what the volume should be, so both columns were being costed against a number that was itself the defect.

Nor was the staff engineer wrong to write the proposal. It was careful work, it produced the infrastructure estimate the review needed, and it was the thing that caused anyone to look at the bill at all. It was aimed at the wrong question, which is a normal outcome when the wrong question is the one everybody is asking.

What remains is to run the loop on something real. Take one layer you are currently paying for or currently building, cost both columns over a stated horizon with operations inside each, write the exit estimate down, check whether the constraint driving any build proposal is genuinely yours, and put the whole thing in a record with three triggers and an owner for each. It takes an afternoon per layer, and the first one you do will almost certainly turn up an assumption that stopped being true some time ago.