AI Dynamics

Global AI News Aggregator

About

LLM Tokenization Trick for EOT Emission

It’s not willing to produce what it thinks is an EOT token. You have to trick it by making it emit a sequence of tokens that re-tokenize to the EOT token after being turned into a string.

→ View original post on X — @goodside