Instrumenting token usage

Prompts grow by accretion — a tool here, a few retrieved chunks there, a provider changing its own serialisation — and none of it is visible unless somebody is recording it. Log per request the effective limit, the predicted and actual input tokens, the fixed overhead, the reserve, the output produced and the provider's stated reason for stopping, then alert on headroom rather than on failures. Every field is an integer or a short string, so the record can be kept at full volume and queried by anyone.

Context window — read the current version at https://daviddominguez.dev/learn/en/context-window/instrumenting-token-usage

Back to top