The more efficient tokenizer is really interesting for non-English languages I wrote a little about how OpenAI's tokenizer penalizes languages like Spanish in this piece about how tokenizers work: https://
simonwillison.net/2023/Jun/8/gpt
-tokenizers/
…

→ View original post on X — @simonw