AI Dynamics

Global AI News Aggregator

About

Model outputs sub-tokens instead of the full token

It can output other tokens that correspond to the same string. It can’t say the token “ SolidGoldMagikarp” but it can say “ Solid”, “Gold”, and “Magikarp”, which renders the same to you.

→ View original post on X — @goodside