AI Dynamics

Global AI News Aggregator

200k Token Vocabulary Optimized for Non-English Languages

By

@swyx

–

200k token vocab, great for non english
https://github.com/openai/tiktoken/commit/9d01e5670ff50eb74cdb96406c7f3d9add0ae2f8
…

→ View original post on X — @swyx,

Commentaires

Leave a Reply

Your email address will not be published. Required fields are marked *

MORE ARTICLES