200k token vocab, great for non english
https://github.com/openai/tiktoken/commit/9d01e5670ff50eb74cdb96406c7f3d9add0ae2f8
…
200k Token Vocabulary Optimized for Non-English Languages
By
–
By
–
200k token vocab, great for non english
https://github.com/openai/tiktoken/commit/9d01e5670ff50eb74cdb96406c7f3d9add0ae2f8
…