I wrote this explainer on tokenizers last year: https://
simonwillison.net/2023/Jun/8/gpt
-tokenizers/
… Effectively they're the vocabulary used by the model

→ View original post on X — @simonw