Three days should buy evidence, and a demo is not evidence

The director's request came with a deadline attached: an AI that reads incoming RFPs — requests for proposal, the long questionnaires a prospective customer sends out before choosing a vendor — and drafts the technical response, and a yes, a no, or something else by Thursday. Two engineers had already opened a branch by the time the meeting ended. Their plan was to take the two most recent RFPs, wire up retrieval over the response archive, generate drafts, and let the quality of the output decide it.

That plan produces an answer to a question nobody asked. Given two RFPs that the team chose, a current model will draft something readable, and everybody in the room knows it will, so the demo has no power to discriminate between the world where this project works and the world where it does not. Worse, the two RFPs will be the two that were easiest to obtain, which means they are structured documents from a familiar sector with questions the archive has answered before. The demo passes, the project is approved, and the first genuine input — a 400-row spreadsheet from a public-sector buyer with per-module pricing and a Spanish-language annexe — arrives in month two.

Seventy-two hours is enough time to buy real evidence about three things, and the answer that comes back on Thursday should rest on those three rather than on a screenshot.

Day one: find the verifier and the corpus#

The first question is whether the archive exists in a form that can produce a labelled set, because everything downstream depends on it. In this case the RFP responses live in a SharePoint folder with 143 documents going back four years. Sixty-one of them were produced in a proposal tool that stored per-question answers, which exports as 4,100 question-and-answer pairs with a section label attached. The remaining eighty-two are monolithic Word documents where questions and answers are interleaved in tables of varying shape, and extracting pairs from those is a parsing project of its own.

Four thousand pairs is a corpus. Zero pairs would have been the end of the investigation on the first afternoon, and it is worth knowing that by lunchtime rather than in week six.

The eighty-two Word documents are worth pricing rather than ignoring, because they are 57% of the archive and the temptation is to assume they can be parsed later. Opening six of them at random shows three distinct table layouts, two documents where the questionnaire arrived as a PDF and the answers were typed into a separate summary, and one where the questions are numbered paragraphs with answers in a different font. A parser for that is not a weekend, and hand-extraction at roughly twenty minutes per document is twenty-seven hours of solutions-engineer time. Neither number is fatal on its own; both belong in the memo, because a project that quietly depends on twenty-seven hours nobody has allocated is a project that stalls in week three.

The second question is who already owns being right. Two solutions engineers write the technical sections; the head of security signs off the security questionnaire; nobody reviews the architecture and integration sections, which go out as written. That asymmetry is the most important finding of the day, because it says the security half of the problem has an inspection point before commitment and the architecture half does not. Any design that survives will have to add one, or restrict itself to the half that already has one.

Day two: size the prize and the damage#

Volume comes from a system rather than from an estimate in a meeting. The CRM records 140 RFPs received in the last twelve months, of which 52 were responded to. Timesheet data and the two solutions engineers agree on a median of 34 engineer-hours per response, so the annual spend is around 1,770 hours, roughly nine tenths of a full-time engineer, or about $150,000 loaded. That is the ceiling on the prize and not the prize; a system that takes 30% out of drafting time saves 530 hours a year, which is real and is not transformative, and the director should hear both numbers.

Error cost comes from history. Searching the archive for commitments that were not kept turns up a 2024 response promising SCIM group provisioning, meaning automatic sync of users and groups from the customer's own identity provider, "available at contract start", which was not built; honouring it consumed six weeks of unplanned engineering on a deal worth $180,000 of annual recurring revenue. One instance is not a rate, and it is proof that the error class exists and that its cost is contractual rather than embarrassing. A generated answer that confidently reuses a past commitment about a feature since deprecated is exactly the failure this system would make most often, because the archive contains four years of statements that were true when written.

The third thing day two buys is a check on demand, and it is the one most likely to change the answer. Asked separately what actually slows a response down, one solutions engineer said drafting and the other said waiting on security review and pricing approval, and the CRM timestamps support the second: median elapsed time from RFP receipt to submission is nineteen days, of which drafting occupies about four. Compressing four days of work inside a nineteen-day cycle does not compress the cycle, which means the value has to be defended as engineer hours saved rather than as deals won faster, and those are different arguments to different people.

Day three: run the inputs you would rather avoid#

The remaining day goes on the eight worst RFPs in the archive, chosen deliberately for the properties that break pipelines: a 400-row Excel questionnaire with yes/no/comment columns, two documents whose questions reference numbered appendices, a Spanish-language public-sector tender, and one that requires separate answers per product module.

The thing to measure is retrieval rather than generation, because generation quality is the part already known to be adequate and retrieval is the part that determines whether the draft is grounded in something true. Build a held-out set from twelve past RFPs where the accepted final answer is known, giving 240 questions, and measure whether the correct past answer appears in the top three retrieved candidates.

def recall_at_k(eval_set, retrieve, k=3) -> dict[str, float]:
    hits, by_section = 0, defaultdict(lambda: [0, 0])
    for item in eval_set:                       # 240 questions from 12 past RFPs
        got = retrieve(item.question, k=k)      # returns answer_ids, ranked
        hit = item.accepted_answer_id in {a.id for a in got}
        hits += hit
        by_section[item.section][0] += hit
        by_section[item.section][1] += 1
    return {
        "overall": hits / len(eval_set),
        **{s: h / n for s, (h, n) in by_section.items()},
    }

# bm25 baseline          -> overall 0.61  security 0.74  architecture 0.52  commercial 0.44
# embeddings + reranker  -> overall 0.84  security 0.93  architecture 0.79  commercial 0.61

Two days of work produced a number that survives contact with a planning meeting, and the per-section breakdown is worth more than the overall figure. Security questions retrieve at 93% because they repeat almost verbatim across tenders; commercial questions retrieve at 61% because they are specific to each buyer's procurement framework and the archive genuinely does not contain the answer. That decomposition is the scope proposal, written by the data rather than by anyone's preference.

What Thursday's memo says#

The memo runs to a page and proposes the third answer. Rather than drafting the response, the system suggests an answer for each question from the past-answer library, with the source response cited and a confidence attached, and abstains below threshold; the solutions engineer edits. Scope is the security and architecture sections, which is 71% of questions by count, and excludes commercial. Success is defined before the build: recall@3 above 0.85 on a refreshed held-out set, and median engineer-hours per response below 22 measured across the next six responses. The known risk is stated with its mitigation — the archive contains commitments that have since become false, so every suggestion carries the date of the response it came from and any suggestion older than eighteen months is flagged amber.

The four conditions that would have made it a no are worth writing into the memo even though none of them fired, because they are what makes the yes falsifiable. If the corpus had been eighty-two Word documents and nothing else, the honest answer is no, since building the per-question corpus is three weeks of solutions-engineer time that nobody will fund and without which there is neither retrieval nor a verifier. If the volume had been fourteen RFPs a year rather than fifty-two, the 480-hour prize would sit below the integration cost. If no reviewer existed anywhere in the flow, the design would be putting contractual commitments in front of buyers with nothing inspecting them, which is chapter 3's unbounded-and-undetectable combination and the strongest no available. And if both solutions engineers had said the bottleneck was pricing approval, the project would be optimising a step that is not the constraint.

The memo is addressed to the director and written so that the two solutions engineers can dispute it, since they are the people whose working day it changes and the ones most able to notice a wrong assumption. It opens with the recommendation rather than the method, because a sponsor reading it at seven in the morning needs the answer in the first two lines, and the evidence is arranged underneath in the order the three days produced it. The success criteria go near the top rather than in an appendix, for the practical reason that criteria buried on page two are criteria that get renegotiated in month three.

What three days is not#

It is not a spike, and the difference matters because a spike answers "can we build it" and this exercise answers "should we". Nothing was built during the seventy-two hours except an evaluation harness and a retrieval baseline, and the harness is the durable artifact: if the project is approved it becomes the eval suite, and if it is rejected it costs a day and a half to rebuild when someone asks again next year.

It is also not a feasibility study in the sense of a document about risks. Every claim in the memo is a number obtained from a system — 140, 52, 34 hours, nineteen days, 4,100 pairs, 0.84 — and the ones that are estimates are labelled as estimates with their source. A sponsor who disagrees with the conclusion can attack a specific number, which is the point of writing it that way, because an argument about whether the median is 34 hours or 22 hours is an argument that converges.

None of the three days went on model selection, and that is deliberate. Which model to use is a decision that takes an afternoon once an eval suite exists, it is reversible, and the answer changes every few months anyway. Spending the discovery window comparing four providers on the two nice RFPs produces a leaderboard that will be stale before the project ships and that says nothing about whether the corpus exists or whether anyone reviews the output.

The same discipline transfers to a request with a different shape and the same clock: the CEO has seen a competitor ship an AI feature and wants parity within a quarter, and your view is that the competitor has shipped a demo. That belief is worth nothing in the room unless you have gone and found the inputs where their feature breaks — which is a specific, boring, entirely doable piece of fieldwork — and it may lose the argument even when you have. What to collect beforehand, and what remains available when the decision goes against you, is the subject of the last chapter.