Trimming the output schema

Not asking for a field is the largest latency lever on a generation endpoint, because those tokens are never generated; generating them and stripping them server-side saves the response body and nothing else. The catch is that the skipped tokens were part of the computation that produced the fields after them, so the cheaper request can return a different and sometimes worse answer — and a schema that declares its score before its reasoning gets the cost of both with the quality of neither.

Autoregressive generation — read the current version at https://daviddominguez.dev/learn/en/autoregressive-generation/trimming-the-output-schema

Back to top