ICYMI – Andrej Karpathy has released an excellent video tutorial on "Tokenization" couple of days back. ⦿ Basics covered: Strings, Unicode code points, and encodings like UTF-8.
⦿ Byte pair encoding algorithm explained and implemented in Python.
⦿ Delving into complexities:

→ View original post on X — @sudalairajkumar,