Why would they do that though? The state of the art in tokenizers has moved on enormously since 2019, why go back to an old design?
Tokenizer Design Evolution: Why Revisit Outdated 2019 Architecture?
By
–
By
–
Why would they do that though? The state of the art in tokenizers has moved on enormously since 2019, why go back to an old design?