AI Dynamics

Global AI News Aggregator

About

Etymology and Tokenization in Language Models Explained

Ah this makes sense. So etymology is baked into the tokenization in this way. I guess there's similarity to phoenetic languages in that sense.

→ View original post on X — @yoheinakajima