It’s whatever the byte-pair encoding algorithm says which depends on the pre-train statistics. There are many shorter ones too — the top 130 longest tokens are all banners and whitespace of various lengths.

→ View original post on X — @goodside