Ah this makes sense. So etymology is baked into the tokenization in this way. I guess there's similarity to phoenetic languages in that sense.
Etymology and Tokenization in Language Models Explained
By
–
By
–
Ah this makes sense. So etymology is baked into the tokenization in this way. I guess there's similarity to phoenetic languages in that sense.