As work to apply our Media Trends analysis reports backwards to the entire quarter-century TV News Archive (no data is used to train or tune any model), we've been trying to achieve the highest possible throughput to complete processing as quickly as possible, while not exceeding the endpoint maximum quota. Model quotas at scale are essentially measured in tokens per minute per model (the reality is a bit more complex, but TPM captures the general gist). Initially we essentially randomly interleaved longer and shorter transcripts and focused on QPS, maintaining a fixed number of inference requests per second and relying on the randomized mixing of long and short transcripts to yield a relatively even TPM rate. However, as the graph on left at the top of the post shows, this yielded an extremely uneven TPM rate that heavily exceeded the endpoint quota and resulted in significant 429 errors. While Gemini offers a dedicated countToken API, given the extreme size of our requests, which each approach the 1M token context window limit, we wanted a lighter-weight approach token estimation. Instead, we wrote an out-of-band monitoring daemon that continually monitors the realtime inference stream from the Vertex AI API Gemini endpoint and compares it with the size of the input transcript (in bytes, characters and words), language and some base complexity metrics and combines to yield an estimated token count formula that we are able to compute entirely on the client side, without requiring a separate API call for token estimation. The end result of this new token estimator can be seen in the graph at top right of this post that shows how we are now able to keep our TPM rate in a very tight band that sits right under the quota threshold, allowing us to maximize our total throughput.