Six engineer-weeks is the smallest term in the comparison

The slide had two numbers on it. On the left, the vendor's quote for an evaluation platform: £80,000 per year, for up to twenty seats and two million scored traces a month. On the right, the team's own estimate for building the equivalent: six weeks. Underneath, someone had done the arithmetic at the fully loaded rate the finance team uses for engineering time, £10,000 per engineer-week, and written £60,000. The CFO looked at £60,000 against £80,000 and asked the obvious question, which was why anyone was still discussing it.

The decision behind that slide is what this topic is about. Build versus buy is the choice, taken component by component, between having your own engineers write and operate a piece of infrastructure and paying a vendor to run their version of it while you integrate against their API. It is not one decision taken once at the start of a project. The layered stack the reference architecture topic laid out — a gateway in front of the providers, orchestration, a vector store, an evaluation plane and an observability plane — poses the same question at every layer, and a team can reasonably answer it differently at each one. The question exists at all because each of those layers is simultaneously a product that several companies sell and a system a competent backend team could write in a quarter, so nothing about the technology settles it. What settles it is cost over a period of years, and most of that cost is the cost of owning a thing after it works rather than the cost of making it work.

You meet the decision in three recognisable situations. A renewal notice arrives with an uplift on it and somebody has to say whether the component is still worth its price. A vendor is proposed and the objection raised is that the company would then depend on them. An engineer proposes replacing a dependency the team keeps working around, on the grounds that building it would be cleaner. Each of those is a later chapter here. This one is the plainest version, where the comparison has been reduced to two numbers on a slide that were produced by different methods.

The component being argued about is an evaluation platform: a hosted service for the eval work an earlier topic covered, which runs a set of scorers over recorded model inputs and outputs, keeps every score alongside the prompt version and model snapshot that produced it, compares one run against another, and exposes the results both to people who do not read Python and to a CI job that can block a release. The team already had scorers; what £80,000 a year buys is somewhere for them to live and a history to compare against. Fully loaded, in the other number, means salary plus employer taxes, benefits, equipment and a share of every other per-head cost the company carries — roughly twice base salary, and the rate finance will insist on.

The answer to the CFO's question is that those two numbers are not the same kind of number. The £80,000 is a price for a year of a running service, including the vendor's on-call rota, their upgrades, their integration work when a provider changes an API, and their support engineer answering a ticket at four in the afternoon. The £60,000 is the cost of reaching feature parity once, on a good week, with the two engineers who designed it still sitting next to each other. Everything the vendor's price includes after the first delivery is missing from the build column, and none of it is optional — it just moves onto your rota instead of appearing on an invoice.

Both columns need the same horizon and the same contents#

The fix is mechanical. Pick a horizon, state it out loud, and put the same categories of cost inside both columns. Three years is usually the right horizon for a piece of platform infrastructure, because it is long enough that maintenance dominates the build spike and short enough that you are not pretending to forecast a market that reprices every six months. Whatever you pick, write it on the page, because a comparison over an unstated horizon is the mechanism by which the six-week figure gets to stand in for forever.

Seven categories account for nearly everything that moves the answer:

• Implementation — the work to reach parity with the capability you are actually buying, not with the demo.
• Integration — wiring it into CI, auth, the data warehouse, and whatever pipeline produces the traces it scores. This one is close to identical in both columns, which is why it is so often left out of both, and leaving it out of both is fine as long as you say so.
• Upgrades — dependency bumps, CVEs (entries in the public catalogue of known security vulnerabilities, each of which obliges you to patch something), the Postgres major version, the SDK that drops a Python release.
• Support — the internal equivalent of the vendor's support engineer, which is one of your engineers answering a Slack message about why a scoring run failed.
• On-call — pages, incident time, and the postmortems that follow.
• Failure impact — what it costs when the thing is down or wrong, which for an evaluation platform is usually a blocked release rather than lost revenue.
• Vendor egress — the cost of getting your data and your instrumentation out at the end, which chapter 3 takes apart properly.

Before any of those categories can be filled in, the two columns have to be describing the same product, and that is where the six-week estimate usually starts going wrong. "An equivalent" is doing a great deal of work in the sentence "we could build an equivalent internally in six weeks", because the vendor's platform has a rubric editor, a diffing view across prompt versions, a Python SDK, role-based access for the twenty seats, retention of scored traces for thirteen months, and an API that CI can block a release on. The team's six weeks covers the scoring loop, a Postgres schema, and a CLI. Both of those are defensible products; they are not the same product, and the honest way to handle it is to write a parity list before costing anything.

For this decision the list came out at seven items: run a named rubric over a named dataset; store per-item scores with the prompt version and model snapshot that produced them; compare two runs and show which items moved; fail a CI job on a threshold; let a non-engineer read the results; retain a year of history; and be usable by twenty people without an engineer in the loop. The team's six weeks covered the first four. Items five through seven are the ones that turn a scoring script into a platform, and they are also the ones that arrive as a stream of small requests over the following year rather than as a project — which is exactly how a build that finished on time acquires another eight engineer-weeks nobody planned.

For the evaluation platform on that slide, filled in with the team's own numbers rather than industry averages, the three-year picture looked like this.

Failure impact is absent from the table because for this component it is small and roughly symmetric: when the evaluation platform is unavailable, releases queue rather than customers suffering, and half a day of blocked deployment costs the team a few thousand pounds of delay in either column. That is worth writing as a line with a zero next to it rather than omitting silently, because the reader of the table then knows it was considered. For a gateway the same line would be the largest number on the page, since the failure mode there is customer requests failing.

£243,000 against £279,200 is a different conversation from £60,000 against £80,000. The gap is about 15%, which is inside the error bars of every estimate in the left-hand column, and that is the useful conclusion: on these numbers, cost does not decide it. What decides it is which column's error bars you trust, and they are not symmetric. The right-hand column contains one number the vendor is contractually bound to, plus a small internal figure. The left-hand column contains eight estimates your own team produced, and the largest of them is the one with the least evidence behind it.

The multiplier you already have data for#

Every team knows, roughly, how its infrastructure estimates land. If the last three internal platform projects took 1.8 times their original estimate, that ratio is a measured property of your organisation and there is no reason to assume this project is the one that behaves. Applying it is a one-line change to the model, and it moves the recurring costs too, because a system that took eleven weeks to build is a bigger system to maintain than one that took six.

from dataclasses import dataclass

WEEK = 10_000  # fully loaded engineer-week, GBP

@dataclass
class BuildCase:
    parity_weeks: float
    annual_maintenance_weeks: float   # provider work + upgrades + on-call + support
    ramp_weeks: float                 # onboarding engineers who did not design it
    annual_infra: float

    def total(self, years: int, estimate_multiplier: float = 1.0) -> float:
        parity = self.parity_weeks * estimate_multiplier * WEEK
        # maintenance scales with the system you actually ended up with
        maint = self.annual_maintenance_weeks * estimate_multiplier * WEEK * years
        return parity + maint + self.ramp_weeks * WEEK + self.annual_infra * years

eval_build = BuildCase(
    parity_weeks=6,
    annual_maintenance_weeks=12.6,   # 12d provider + 4d upgrades + 5d on-call, in weeks
    ramp_weeks=2,
    annual_infra=7_200,
)

for m in (1.0, 1.4, 1.8):
    print(f"x{m}: £{eval_build.total(years=3, estimate_multiplier=m):,.0f}")

x1.0: £243,600
x1.4: £328,440
x1.8: £413,280

At the team's own historical accuracy, the build costs half again what the licence costs, and the decision stops being close. Presenting the multiplier explicitly is also what keeps the conversation honest, because the alternative — arguing about whether it is really six weeks — has no end. Nobody has to concede that their estimate is wrong; they only have to accept that this project will behave like the last three.

There is a matching sensitivity on the buy side, and it belongs on the page too. Vendor pricing for evaluation platforms is usually per-seat or per-trace, and both scale with success. If the trace volume triples because the product works, the £80,000 becomes something else, and the contract's overage rate — what you are charged per seat or per thousand traces once the bundled allowance is used up, rather than its headline — is the number to model. Ask for it before signing, and put the three-year figure at your forecast volume in the table rather than at today's.

The line item everyone leaves out#

The one that disappears most reliably is the second engineer. The build case is nearly always costed with the designer of the system as its permanent maintainer, and that assumption survives about eighteen months in most organisations. When the original author moves to another team, someone who has never seen the code inherits an internal platform with no documentation beyond its tests, no community to search, no changelog, and no Stack Overflow. Two weeks of half-productive ramp is a conservative figure for that, and it recurs every time ownership changes hands. A bought component has the same problem in a much weaker form, because the new engineer can read the vendor's documentation at two in the morning without waking anyone.

Closely behind it is the displaced product work. Six engineer-weeks spent on an evaluation platform are six engineer-weeks not spent on the thing customers pay for, and the £10,000 rate is a cost proxy rather than a measure of that opportunity. Two cautions apply. Do not count it twice — if £60,000 is already in the table as cost, adding a separate "opportunity cost" line double-counts the same weeks. And do not inflate it with a speculative revenue figure, because a number you invented will be the first thing attacked and it will take the credible parts of your analysis down with it. The honest form is a sentence next to the table naming what those six weeks were going to be spent on, so that the person deciding can see the trade rather than a made-up multiple.

What this model does not settle#

A total-cost model is not a decision procedure. It produced £243,000 against £279,200, and if you treat a 15% gap as a verdict you have mistaken the precision of the arithmetic for the precision of its inputs. What the model does is make the assumptions legible — the horizon, the volume, the maintenance rate, the estimate multiplier — so that in eighteen months someone can check each one against what actually happened. Chapter 5 turns that into a written record; chapter 6 is what happens when nobody wrote one.

It is also not a discounted cash flow, and resisting the urge to build one is worth a moment. At these amounts and this horizon, discounting moves the totals by a few per cent and changes no ranking, while adding a dial that anyone can turn to get the answer they wanted. If the finance team asks for NPV — net present value, the same cash flows with later years discounted on the principle that money spent in three years costs less than money spent today — give them the cash flows by year and let them apply their own rate.

There is one genuinely non-engineering force worth naming, because it will show up and it is legitimate. Internal build effort can often be capitalised as an asset and amortised — recorded as something the company now owns and charged against profit a slice at a time over several years — while a licence is an operating expense, charged in full in the year it is incurred. A CFO may therefore prefer the build for reasons that have nothing to do with whether it is a good system, and engineers who do not know this happens tend to conclude that the finance team is being irrational. It is a real constraint, it belongs in the conversation with its name attached, and it does not change the engineering cost of owning the thing — only which column of the accounts that cost lands in.

Finally, the model says nothing about capability. A vendor evaluation platform that scores traces against rubrics you cannot express is not cheaper than building; it is not the same product. Costing two things that do different jobs is the most confident way to get a precise wrong answer, so the first step is always to write down what parity means and check that both columns clear it.

The evaluation platform was, in one respect, an easy case: a vendor existed, they sent a quote, and there was a number to argue with. The layer where this analysis goes wrong most often is the one where nobody bothers to get a quote at all, because every backend engineer who looks at it is confident they could build it in a weekend. That layer is the gateway, and the next chapter costs the version of it you end up owning rather than the one you demo.