TIL the OpenAI tikoken library rejects text that includes special tokens by default, which affects my own "ttok" CLI tool – it raises an error if you call it like this: ttok '' –encode

→ View original post on X — @simonw