Cancelling a stream

Billing follows compute performed. Cancel after 300 of an expected 800 tokens and you pay for 300 forward passes that cannot be un-run, while saving the 500 that never happened and the nine seconds of accelerator time they would have taken. Cancellation cannot skip ahead either — no path through the network produces token 750 without tokens 1 through 749 — and the usage event never arrives on an aborted stream, so the tokens have to be counted as they land.

Autoregressive generation — read the current version at https://daviddominguez.dev/learn/en/autoregressive-generation/cancelling-a-stream

Back to top