The completion check at the response boundary

An extraction service that writes 0.4% of its rows with a field the model never finished producing has no broken component. The HTTP client, the SDK, the JSON parser, the schema validator, the business rule and the database column each behave exactly as specified, and a review that goes looking for the faulty one will not find it.

What that service is missing is a component rather than a fix: a single place where somebody asks whether generation completed. This chapter is about where that place is, why it can only be there, how to test it, and what belongs beside it so that the condition is visible even when the check is doing its job.

Every layer answered its own question correctly#

Walk the path a truncated response takes through a typical service and the absence becomes structural rather than accidental.

The HTTP client received a 200 and returned it. The SDK deserialised the body into a message object and did not raise, because a stop reason is data, not an error — raising on max_tokens would break every caller who deliberately caps output for a preview. The JSON parser was handed valid JSON and produced a dictionary. The schema validator was handed an object with all its required fields and produced a domain object. The business rule that follows was asked whether an empty notes is acceptable and said yes, correctly, because 8% of invoices have no notes. The database column was nullable, correctly, for the same reason. Monitoring tracked error rate, p95 latency and rows written per hour, all of which were normal, because 28 truncated responses per day on 7,000 requests is 0.4% and every one of them produced a row.

Reviews go badly when this is not stated plainly. Every layer in that chain answers a question correctly, and none of the questions is "did generation complete?" — so the system never asks it, and a system that never asks a question cannot be said to have got the answer wrong. Reviews that go looking for the broken layer either find nothing, which produces a shrug and no action, or settle on the most proximate layer and add a rule there, which produces a field-level validator and its slow decay into carve-outs.

The belief that stalls those reviews is that a bad outcome implies a broken component. It is a good belief most of the time, which is what makes it hard to put down: the large majority of production faults really are one component doing something other than what it claims, and the technique that follows from the belief — narrow the blast radius, find the layer, read its code — genuinely works. It does not work here, and the reason is that correctness of a part is defined against the question that part was given. A layer answering its own question correctly is correct, and remains correct however bad the outcome is. A chain of such layers is complete only up to the set of questions somebody thought to ask, and no member of the chain is answerable for a question that was never written down. What is absent is not behaviour inside a layer. It is a layer, which is why the fix reads as an addition, and why hunting for the culprit ends in a shrug.

The dotted edge marks the only span in that diagram where the check can be written, and the reason is the label on node E. The finish reason lives on the provider's response object. The moment that response becomes an Extraction, the field is gone — not hidden, not inconvenient to reach, gone, because your domain type does not have a slot for it and should not. Every layer downstream of that conversion is working with an object that no longer carries any evidence about how the generation ended, which is why none of them could have caught this no matter how carefully they were written.

Stated as a rule, that gives a placement argument you can apply to checks with nothing to do with truncation. A check needs two things in scope at the same time: the evidence it examines, and the knowledge of what the right answer is here. The finish reason is in scope from the moment the SDK parses the response until the moment the domain object is built, and nowhere afterwards. What counts as an acceptable ending is in scope only where the purpose of the call is known — an agent loop that expects tool_use, a preview that means its cap. Above the call site the first is available and the second is not. Below the conversion the second is available and the first is gone. Those two intervals overlap in exactly one region, and the conversion function sits at the end of it. The boundary is therefore not a matter of taste about layering. It is the last point at which both of the facts this check requires are in hand at once, and a check placed anywhere else has to reconstruct one of them.

Put the check where the information still exists#

The conversion from provider response to domain object is where a codebase stops thinking in terms of the API and starts thinking in terms of the business. That is the last place the finish reason exists and the first place anyone would think to look for it, which makes it the right place for one reason: it is the point of no return for the information.

class IncompleteGeneration(Exception):
    def __init__(self, stop_reason: str | None):
        super().__init__(f"generation did not complete: {stop_reason}")
        self.stop_reason = stop_reason


def to_extraction(response) -> Extraction:
    """The single boundary between the provider's response and our domain."""
    stop_metric.labels(reason=response.stop_reason or "none").inc()

    if response.stop_reason not in ACCEPTABLE_ENDINGS:
        raise IncompleteGeneration(response.stop_reason)

    return Extraction.model_validate_json(response.content[0].text)

Three properties make this hold up where a field rule does not. It is applied to the finish reason rather than to any field, so it catches truncation in notes, in line_items, in vendor_address, and in the field somebody adds next quarter, without being edited. It has no opinion about what an empty string means, so legitimately empty notes flow through untouched and the false-failure problem never arises. And it is one function, so there is one place to read to know what this service considers a complete response, rather than a set of field validators that collectively imply an answer.

The check runs before model_validate_json, and the ordering is the same rule that governs repair passes: the cheap, definitive check about whether the response is worth interpreting comes before any interpretation. Reversed, the validator would sometimes raise its own error first — a type error on a half-emitted number, say — and the diagnostic you get would be about a schema mismatch rather than about a truncation, which sends whoever is on call to the wrong file.

A check like this is cheap to test, which matters because a check nobody tests is a check that quietly stops working the day someone refactors the boundary. You do not need the network for it — fabricate the response shape and assert the behaviour:

def test_truncated_response_is_rejected():
    response = FakeResponse(
        stop_reason="max_tokens",
        text='{"id": "A-1", "total": 4200, "notes": ""}',   # valid JSON
    )
    with pytest.raises(IncompleteGeneration):
        to_extraction(response)


def test_legitimately_empty_notes_is_accepted():
    response = FakeResponse(
        stop_reason="end_turn",
        text='{"id": "A-2", "total": 900, "notes": ""}',
    )
    assert to_extraction(response).notes == ""

The pair matters more than either test alone. The first pins the behaviour a truncation bug is about; the second pins the behaviour a field validator gets wrong, and it is the one that stops a future contributor from "improving" the check into a non-empty-notes rule.

The metric that belongs beside the exception#

Alerting on the rate of non-natural finish reasons is not a substitute for the check, but it is a necessary complement, and it is the second half of what a service in this state is missing. The counter in to_extraction costs one line and turns an invisible 0.4% into a labelled series. At 7,000 requests a day, 28 truncations do not move an error-rate panel that renders in whole percentages, but they are unmissable on a graph of stop reasons by count, and a threshold on that graph fires on the day a model migration or a longer document class pushes the rate from 0.4% to 4%. The exception stops the bad row from being written; the metric tells you it is happening at all. Neither one does the other's job.

The invisibility is worth doing as arithmetic, because the arithmetic is what shows that nobody involved was careless. Twenty-eight responses out of seven thousand is four in every thousand, which is the 0.4% above. A dashboard that renders an error rate to the nearest whole percentage point displays four in a thousand as zero. It also displays it as zero on the day the rate doubles, and on the day it doubles again. Nothing in that panel is broken; a percentage rounded to a unit cannot represent a quantity living two orders of magnitude below the unit, and no amount of attention paid to the panel changes what it is able to show. Counting stop reasons separately escapes the problem because a count of twenty-eight is a count of twenty-eight regardless of what the denominator is doing, and because a step change in a small count is plainly visible on a graph of that count while being invisible in a ratio that was rounding to zero on either side of the step.

The metric's design is unfussy and worth getting right on the first pass. Label by stop reason and by call site, and by nothing else — the reason field has a handful of values and your call sites are countable, so cardinality (the number of distinct label combinations, and therefore of separate time series, that the metric creates) stays trivial, whereas a label carrying the model string or the document type will grow without bound and make the series expensive. Count every response, including the successful ones, because a rate needs a denominator and reconstructing one from a separate request counter is the kind of thing that goes subtly wrong during a deploy. Alert on the ratio rather than the raw count so that the threshold survives traffic growth.

Choose the threshold from the measured baseline rather than from zero. A service that legitimately truncates 0.3% of the time because a small share of its inputs are genuinely enormous will page somebody every night under a zero-tolerance alert, and an alert that pages every night is an alert that gets muted. Setting it at three to five times the observed baseline catches the step changes — a migration, a new document source, a prompt edit that made answers longer — which is what you actually want to know about.

Why the layers above and below cannot host it#

Pushing the check up, into an SDK interceptor or an HTTP middleware that raises on any non-end_turn response, is appealing because it would cover every call site at once. It fails on the same point that makes an allowlist a per-call-site constant: which endings are acceptable is a property of the call site, not of the transport. An agent loop must accept tool_use; a call using server-side tools must accept pause_turn and resume; a deliberate preview request may accept max_tokens and mean it. A middleware that raises on all of them breaks working code, and one that accepts all of them checks nothing. Hoisting the check above the point where those distinctions exist requires throwing away the distinctions.

Pushing it down, into schema validation, is closed off for a different reason. Schemas describe documents, and interruption is not a property of a document.

An exception is also a decision to stop rather than a recovery, and conflating the two produces the retry loop that re-runs the identical request against the identical cap and truncates three times before giving up. A truncation is deterministic given the same inputs, so a useful retry has to change something. Re-issuing with a substantially larger max_tokens works when the cap was simply undersized. Splitting the document and extracting in two passes works when the input is the outlier. Routing to a dead-letter queue for manual handling works when neither is safe to do automatically. All three are choices made by the caller with knowledge of the workload, which is precisely why the boundary function should surface the condition and stop rather than guess.

Nor is the check a correctness gate. A response with stop_reason: "end_turn" and a hallucinated total passes it, as it should. There is even a case it cannot see: a model that decides it is finished while having answered half the question emits a perfectly natural ending on a genuinely incomplete answer, and no finish-reason check will ever catch that, because from the API's point of view nothing went wrong. That case belongs to prompting and evaluation, and it is worth naming precisely so that the completion check does not get blamed for missing something it never claimed.

What the boundary buys is narrow and durable. One function, one comparison, one counter, placed at the last point where the provider's account of the generation still exists — and after that point, every downstream layer is free to reason about documents without ever needing to know that a model produced them. Systems that lack it are not systems whose components are wrong; they are systems in which the question was never asked, which is why the fix is an addition rather than a repair.

That failure came from a bound set too low. The remaining case is stranger, and more common in user-facing products: a bound set exactly where you intended, doing exactly what you asked, and still cutting the answer in half.