Being overruled is a position to hold, not a defeat to absorb

The CEO saw it on a conference stage in March: a competitor's product answering questions about a customer's own document library, on screen, in front of four hundred people. The instruction that came back was parity within the quarter. Two engineers in the room believed the feature was a demo with a marketing page attached, and said so, and the meeting did not go their way, because "I think that's a demo" and "I think it works" are two opinions of equal evidentiary weight and only one of them was held by the CEO.

The belief was probably correct and it was worth nothing in the room. A claim about someone else's product is unfalsifiable until somebody goes and uses it, which is a piece of fieldwork that takes about a day and that almost nobody does before the meeting where it matters.

Collect the evidence that makes the claim checkable#

The work is unglamorous. Sign up for the competitor's product on a paid plan under a real company card, load sixty documents drawn from your own corpus rather than from theirs, and run the questions your own customers actually ask. Record what happens, including the times.

That table is a different object from an opinion. Each row names an input, a result, and a source, so the CEO can check any of them personally in ten minutes, which is exactly what makes it persuasive. The page-12 truncation in particular reframes the entire discussion, because it is the difference between a system that reads documents and a system that reads the beginnings of documents, and it is the kind of limit that a conference demo is specifically constructed to hide.

Presenting it well matters as much as collecting it. The finding to lead with is the one that is cheapest for the other person to verify and hardest to argue with, which here is the vendor's own beta warning and their own documentation about permissions, because those are their words rather than your measurements. Latency and truncation come second. What not to do is present the table as proof that the competitor is incompetent, since the same table describes a reasonable early release and the argument you actually want is about what parity should mean.

The version of this argument you can lose while being right#

There is a case where every row of that table is true and the conclusion still does not follow, and thinking it through before the meeting is what stops the evidence from being wasted. If the competitor's feature is winning deals or defining what buyers now expect, then its engineering quality is beside the point, because the problem in front of the company is the deal rather than their code. Ask sales for a count: in how many of the last sixty qualified opportunities did a buyer raise it, and in how many did it change the outcome. Two out of sixty is one situation and nineteen out of sixty is another, and being right about the truncation bug does not help in the second.

That count also opens the cheaper answers. A sales objection is often answerable with a roadmap statement, a design partner programme in which a few named accounts get early access in exchange for feedback, or a narrow capability shipped to three accounts, and any of those can be delivered in a quarter without committing a team to matching a feature that the competitor may quietly retire. Offering one of these is a stronger move than asking for two quarters instead of one, because a schedule request concedes the scope and argues about the calendar, which leaves the same unverifiable system on the roadmap sixty days later.

Four things that remain available after you lose#

Sometimes the decision goes ahead regardless, and the mistake worth avoiding is treating that as the end of the engineering conversation. Four moves stay open, they cost days rather than weeks, and they are considerably more valuable arranged before the build starts than assembled afterwards.

Narrowing is chapter 5's move applied under duress rather than in good conditions. Parity with a four-capability feature becomes the one capability where an inspection point already exists in your own flow — for this company, answering support-agent questions from the internal knowledge base, where the agent reads the answer before sending it and the send is the inspection. The other three go on a list with the evidence about what they would require, so that the scope reduction is a documented engineering position rather than a quiet omission.

Review points and a spend ceiling have to be agreed in the same conversation that approves the work, because a ceiling proposed in week seven reads as sabotage and a ceiling proposed in week zero reads as budgeting. Both numbers should come from the sizing arithmetic rather than from your opinion of the project, which is a real trap: a ceiling set deliberately low starves the build and then produces evidence of failure that the mechanism itself caused, and everyone involved will eventually work out that is what happened.

mandate: "KB answering for support agents (slice 1 of 4)"
ceiling:
  inference_usd_per_quarter: 45000     # 3x the modelled steady-state from the sizing memo
  alert_at_pct: 70
reviews:
  - date: 2026-05-18                   # week 6
    metrics:
      answer_used_unedited_rate: {threshold: 0.35, direction: above}
      abstention_rate:           {threshold: 0.40, direction: below}
  - date: 2026-06-22                   # week 11
    metrics:
      handle_time_delta_seconds: {threshold: -25, direction: below}
      escalation_rate_delta:     {threshold: 0.00, direction: below}
decision_rule: "Two consecutive misses on the same metric ends the slice."
agreed_by: [ceo, vp-eng, support-director]

The two metrics chosen at week six are deliberately about behaviour rather than about model quality, because model quality is the thing everyone will want to renegotiate. Whether agents send the answer without editing it, and how often the system declines to answer, are facts about the world that do not depend on anyone's judgement of a generated paragraph.

Instrumentation is what makes those facts available, and it is the move most often skipped because it produces nothing a user can see. The requirement is that every output leaves a record which can be joined to what happened next.

create table kb_answer_events (
  event_id       uuid primary key,
  ts             timestamptz not null,
  question_hash  bytea       not null,          -- dedupe without storing PII
  model_id       text        not null,
  prompt_version text        not null,
  retrieved_doc_ids text[]   not null,
  abstained      boolean     not null,
  agent_id       text,
  sent_verbatim  boolean,                       -- null until the agent acts
  edit_distance  int,                           -- chars changed before sending
  ticket_id      text,                          -- joins to reopen + CSAT later
  latency_ms     int         not null,
  cost_usd       numeric(10,6) not null
);

Without sent_verbatim, edit_distance and ticket_id, the week-six review is a conversation about impressions, and a feature that nobody uses will survive it on the strength of two enthusiastic anecdotes. With them, the review takes twenty minutes and produces a number that the CEO can act on. The instrumentation is not a hedge against the project; it is the part of building it well that makes the difference between shipping a feature and shipping a rumour.

The record of disagreement is the smallest artifact and the one most often written wrongly. What makes it a record rather than a protest is that it states a prediction with a date, a metric and a threshold, and it states what would show the prediction to be wrong. An engineer who writes "I believe this will fail" has produced a feeling with a timestamp. An engineer who writes "I expect the unedited-send rate to sit below 20% at week six, because the knowledge base has 3,400 articles of which 40% are more than two years stale; if it lands above 35% I was wrong about the corpus" has produced something that settles, in either direction, on a known date, and that both parties can sign.

The record has to be able to prove you wrong#

At this company it did. The prediction was an abstention rate above 40% because of the stale knowledge base; the measured rate at week six was 9%, because agents were asking a narrower band of questions than the corpus audit suggested and the stale articles were concentrated in areas nobody asked about. Unedited sends came in at 51% against a predicted 20%. The record was re-signed with the prediction marked wrong and one line explaining why the corpus audit had misled — it measured article age rather than article traffic — and the second slice was approved on the strength of the week-six numbers.

That outcome is what the whole mechanism is for, and a record that could only ever vindicate its author is a protest wearing a document's clothes. The check to run before signing one is whether it contains a condition under which you concede. If it does not, it is not going to be read as a technical position by anybody, and it will make the next rejection harder to get heard.

None of this changes the base rate: most of these arguments are decided by people who own the budget, and a fair number of them go against the engineer who did the fieldwork. What the four moves buy is that losing costs a slice, a quarter and a measured outcome rather than a team and a year, and that the decision comes back around on a date somebody already agreed to.

The proposals to start with are the ones already in flight. Take whatever is on the roadmap now, and for each of them write down what a wrong output would be checked against, what a wrong output costs and who would notice, what the deterministic version scores, and what the narrowed version would be. Two of them will need a rejection with a blocking condition and a date, and one of them will turn out to be the problem next door that nobody had written down. The field report What We've Learned From A Year of Building with LLMs is worth reading alongside that exercise, since it is largely a catalogue of teams discovering these same filters after building the thing first.