How do you add new languages to a model? Efficient tokenization plays a major role here. In this new blog post, we talk about how we expanded LFM2's tokenizer to add new languages.
By
–

How do you add new languages to a model? Efficient tokenization plays a major role here. In this new blog post, we talk about how we expanded LFM2's tokenizer to add new languages.