AI Dynamics

Global AI News Aggregator

About

Tokenization limits: variable tokens, cannot handle per-letter tasks

Tokenization. It doesn’t see text as sequences of characters, but of variable-length tokens about 4 chars on average. It generally can’t do anything that needs to be done one letter at a time. It’s also bad at counting in its head.

→ View original post on X — @goodside