@OpenAI also released a new tokenizer This makes it possible to more accurately (and quickly) estimate your token usage This is actually extremely helpful for the `collapse` step mentioned above!
OpenAI Releases New Tokenizer for Accurate Token Usage Estimation
By
–