atm nanochat does not pretokenize, instead tokenization is done live during training in the dataloader, which is ok and gives the CPU something to do while the GPUs are working. So I didn't try too hard to optimize this part.

→ View original post on X — @karpathy