The seam belongs around a task, with its prompt, schema and evals

The rewrite that followed the failed Tuesday started from a question the incident review kept returning to: how many different things does this platform ask a model to do? Not how many call sites, which was forty, and not how many modules, which was twenty-three. The answer, after two people spent a morning reading the forty prompts side by side, was nine. Six of the call sites were the invoice extractor invoked from different entry points with the same prompt. Eleven were the ticket router, duplicated because two teams had each written their own copy and the copies had drifted — one of them still listed a category that had been removed from the product in March.

Nine is a number you can plan a migration around. Forty is a number you can only be frightened by, and twenty-three modules is a number that tells you about your package layout. The difference between the three is not arithmetic; it is that "nine tasks" names units that can each be migrated, tested and signed off independently, and the other two name units that cannot.

What a task owns#

A task is one job you ask a model to do, with a stable input type and a stable output type that the rest of the system depends on. classify-ticket takes a ticket body and returns one of four categories. extract-invoice takes a document and returns an Invoice. Those contracts belong to the application and do not change when the provider does. Everything between them is provider-specific and should live inside the task, which is the inversion of what the previous chapter found scattered across call sites.

# tasks/base.py
class Task(Protocol[In, Out]):
    name: str
    version: str

    def run(self, payload: In) -> Result[Out]:
        """Execute against the currently bound provider binding."""

@dataclass(frozen=True)
class Binding:
    """Everything provider-specific for one task on one provider."""
    provider: str
    model_id: str
    render_prompt: Callable[[In], Request]   # messages, system, tools, response_format
    parse: Callable[[RawResponse], Out]      # including repair rules
    retry: RetryPolicy                       # what is retryable *for this task*
    params: dict[str, object]                # temperature, max_tokens, reasoning effort

A binding, in the sense the rest of this topic uses the word, is that second dataclass: one task's provider-specific half, holding the model identifier, the code that renders a request, the code that parses the reply, and the parameters the pair were tuned with. A task can carry several bindings at once, and which one serves traffic is a runtime choice. The unfamiliar part for most teams is that render_prompt returns a whole request rather than a string. That is deliberate, because the choice between putting a schema in the prompt text and putting it in a response_format field is itself provider-specific, and a boundary that can only carry a string forces every provider into the first option. Inside render_prompt the task is free to use whatever the provider gives it.

# tasks/extract_invoice/anthropic.py
def render_prompt(doc: Document) -> Request:
    return Request(
        system=SYSTEM,                      # 340 tokens, cacheable prefix
        messages=[{"role": "user", "content": doc.text}],
        tools=[INVOICE_TOOL],               # schema as a tool, forced
        tool_choice={"type": "tool", "name": "emit_invoice"},
        params={"temperature": 0.0, "max_tokens": 2048},
    )

def parse(raw: RawResponse) -> Invoice:
    block = raw.tool_use("emit_invoice")    # no fence stripping; there are no fences
    return Invoice.model_validate(block.input)

# tasks/extract_invoice/openai.py
def render_prompt(doc: Document) -> Request:
    return Request(
        system=SYSTEM_WITH_FORMAT_NOTE,     # 390 tokens; adds a date-format caveat
        messages=[{"role": "user", "content": doc.text}],
        response_format={"type": "json_schema", "json_schema": INVOICE_SCHEMA,
                         "strict": True},
        params={"temperature": 0.0, "max_tokens": 2048},
    )

def parse(raw: RawResponse) -> Invoice:
    return Invoice.model_validate_json(raw.text)

The two bindings are not the same shape, and they should not be. One uses a forced tool call, where the request declares a tool with a typed argument list and names it in tool_choice, so the model's reply arrives as arguments to that tool instead of as prose that has to be found and unwrapped. The other uses a strict JSON schema, the structured-output mechanism from chapter 1, where the provider constrains the text it emits to validate against the schema in response_format. The system prompts differ by fifty tokens because one of the two models needed an explicit instruction that issue_date must be normalised to ISO 8601 rather than copied verbatim from the document. Those divergences used to be invisible; here they are two files in one directory, and the diff between them is a readable statement of how the providers differ for this job.

What stays outside is the contract the caller sees. extract(doc) -> Invoice is the same function it always was, called from the same six places, and nothing in billing/ knows which binding is active.

The eval set is part of the task, not a parallel artefact#

The reason nine is a plannable number and forty is not is that each of the nine can carry its own evidence. Evaluation in CI, from topic 3.16, gives the mechanism; what changes here is where the eval set lives. It sits in the task directory, next to the two bindings, and it is the thing that says whether a binding is fit to serve.

# tasks/extract_invoice/eval.py
CASES = load_jsonl("cases.jsonl")           # 240 invoices, 31 from incident tickets

GATES = [
    Gate("schema_valid",     lambda r: r.parse_ok,                  threshold=1.000),
    Gate("field_exact",      field_match_rate,                      threshold=0.960),
    Gate("date_iso8601",     lambda r: iso_date_rate(r) ,           threshold=0.995),
    Gate("total_within_1c",  total_match_rate,                      threshold=0.990),
    Gate("p95_latency_ms",   p95_latency,                           threshold=4200, lower_is_better=True),
    Gate("cost_per_1k_usd",  cost_per_thousand,                     threshold=9.00,  lower_is_better=True),
]

Two hundred and forty cases is not a research benchmark, and it does not need to be. Thirty-one of them came from production incidents, which is the part that carries the weight, because those are the invoices that broke something once and are therefore the ones a new binding is most likely to break again. The gates include latency and cost, which matters because a binding that passes on accuracy and costs 2.4x is a failed migration even though every quality number is green.

Finding the task boundaries in code that already exists#

Nobody starts with nine tasks written down. The morning of reading forty prompts side by side produced them, and the rule that did the work was that two call sites belong to the same task when they have the same input contract, the same output contract, and would be judged by the same eval set. Everything else — which service calls it, which team owns the caller, whether the prompt strings are byte-identical — is irrelevant to the grouping.

Applying that rule surfaced three patterns worth naming, because they recur.

The first is straightforward duplication with drift. Eleven call sites were the ticket router, and the two copies had diverged: one listed four categories, the other listed five, including partnerships, which the product had removed in March. Traffic routed through the second copy had been producing an invalid category at a low rate for four months, absorbed by a downstream else branch that filed anything unrecognised under other. Collapsing the eleven into one task did not merely reduce the migration surface; it fixed a bug that no monitoring had been positioned to see.

The second is a prompt doing two jobs. One call site asked the model to both classify a ticket and, if the classification was billing, extract the invoice number. That is two contracts wearing one prompt, and it resisted grouping until it was split, at which point the classification half merged into classify-ticket and the extraction half turned out to be a three-line special case of extract-entities. Prompts that resist grouping are usually prompts doing more than one thing, and the resistance is the signal.

The third is a genuine near-miss that should stay separate. summarise-thread and draft-reply both take a ticket thread and both produce prose, so a first pass merged them. They came apart again as soon as anyone tried to write one eval set, because a summary is judged on coverage and faithfulness while a reply is judged on tone, accuracy and whether it commits the company to anything. Two contracts that need different gates are two tasks, and the eval set is the arbiter of that question rather than the prompt's shape.

The output of the morning was a directory per task, each holding a contract, one or more bindings, a case file and a gate list, plus a thin adapter at every old call site delegating to it. That refactor took nine days and produced no user-visible change, which is a hard thing to fund and was funded only because a production incident had just made the alternative concrete.

A swap becomes nine independent decisions#

With the seam at the task, the migration stops being an event and becomes a queue. Each task has two bindings, a suite that runs both against the same cases, and a flag that says which binding serves production traffic. The order of operations is the same for every task, which is what makes it schedulable.

The transition worth studying is Failing --> Deferred. A task that will not qualify does not block the other eight, because nothing about the migration was all-or-nothing. When this platform ran the exercise for real over three weeks, six of the nine tasks qualified within a day each and moved to shadow traffic, meaning the candidate binding ran on live inputs alongside the binding actually serving, with its output recorded for comparison and then thrown away; two needed prompt rework, one of them substantial; and summarise-thread was deferred indefinitely, because the candidate binding produced summaries that eight of ten reviewers preferred while running 3.1x the cost per call, and nobody could justify that for a weekly digest. That task still runs today on the incumbent, which is the term used from here on for whichever binding is currently serving production traffic, and the system is not broken by having two providers in it. It is running two bindings, which is what the seam was for.

Blast radius you can count before you start#

The thing the task seam buys is that the size of a migration is knowable in advance. Before the rewrite, the honest answer to "what does it cost to change provider?" was that nobody could say without reading the code, which is why the estimate came in at half a day. Afterwards the estimate is assembled from a table that already exists.

The nine rows are the plan. The research-agent row is expensive for a reason that is legible from the table itself, which is that it carries seven tool definitions whose descriptions were tuned against one model's tool-selection behaviour, and retuning them is genuinely a week. That number was available before anyone started, and a week of known work is a different conversation with a director than an open-ended risk.

What this does not give you#

It does not give you a provider-agnostic prompt, and the design does not want one. There are two prompts for extract-invoice and they are allowed to diverge as far as the evidence says they should, because the eval set is what enforces sameness, at the level of behaviour rather than text. A team that tries to keep one prompt working well on two providers is doing continuous double-tuning and will end up with a prompt that is second-best on both.

It does not make the swap free, either. Six qualifying on day one looks cheap, and the six were cheap because someone had already built the eval sets — 240 invoices with ground-truth fields, 900 labelled tickets, 1,100 ranked search pairs. That labelling is the cost, and a team that has never written an eval set will find that the task seam converts an unbounded migration risk into a bounded and quite large data-labelling project. That trade is usually worth taking, and it is a trade rather than a saving.

There is also a boundary case the seam handles poorly, which is a task whose shape differs between providers rather than whose wording does. A research agent built around one provider's server-side tool execution, where the provider runs the loop, does not have a counterpart binding elsewhere; the candidate binding has to reimplement the loop in your own process. The interface still holds, in that run(payload) -> Result[Out] is satisfied by both, and the eval set still adjudicates. What it costs is not a prompt rewrite but a small orchestrator, and pretending that is the same size of job as retuning a classifier is how six-day estimates get made.

Reading the two invoice bindings side by side raises a question the next chapter has to answer. One of them uses a forced tool call and a 340-token cacheable system prefix, cacheable meaning that the provider will store its processed form after the first call and re-serve it to later calls that begin with the same tokens, at a fraction of the input price. The other uses strict JSON schema enforcement. Neither of those exists on all providers, and a natural instinct at this point is to define a single interface that works everywhere by using only what everyone supports. That instinct has a price, it is paid continuously rather than at migration time, and it is often larger than the migration it is meant to insure against.