Awesome, glad to hear! And I am curious about the Julia experiments, but absolutely no rush 🙂
Btw I also did some quick comparisons with some other BPE implementations here: https://
github.com/rasbt/LLMs-fro
m-scratch/blob/main/ch02/02_bonus_bytepair-encoder/compare-bpe-tiktoken.ipynb
…
(I also have my own custom one that I plan to polish up and add some time…)

→ View original post on X — @rasbt