AI Dynamics

Global AI News Aggregator

About

Long tokens exist because they’re common; ChatGPT knows their contents and length.

Not that surprising — the only reason long tokens exist is because they're common. If you ask about long tokens (in ChatGPT Classic so it doesn't use code) it knows what punctuation/words they contain, can guess their approximate length, etc.

→ View original post on X — @goodside