Last night I taught nanochat d32 how to count 'r' in strawberry (or similar variations). I thought this would be a good/fun example of how to add capabilities to nanochat and I wrote up a full guide here: https://
github.com/karpathy/nanoc
hat/discussions/164
… This is done via a new synthetic task
@karpathy
-
Teaching AI Models New Skills: The R-Counting Example
By
–
-
Frontier LLM Production Stack Innovation Discussion
By
–
Ty, I just meant it hasn’t (to my knowledge) created a whole new stage in the production stack of a frontier LLM, or had a contribution on that level.
-
NanoChat Live Tokenization During Training Optimization
By
–
atm nanochat does not pretokenize, instead tokenization is done live during training in the dataloader, which is ok and gives the CPU something to do while the GPUs are working. So I didn't try too hard to optimize this part.
-
Regex bottleneck in tokenizer requires expert optimization
By
–
Yep, the use of regex is both a huge dependency and huge bottleneck in the tokenizer. I think it's a beautiful project to try to do this correctly, but I'd need someone who is really familiar with regex to pitch in and also a large test suite to make sure. I'd love to merge such
-
Learning Error Recovery Without Creating Errors in Testing
By
–
That’s almost separate… You certainly want to learn recoveries in the event of errors at test time. You just don’t want to also learn to actively create errors.
-
GPT-10 Input Architecture: Pixels Over Tokenization
By
–
Haha. I am afraid people interpreted my “delete tokenizer” as “use bytes directly without BPE”, the issue is you *still* need bytes encoding arbitrariness even for that! Pixels is the only way. Just like humans. It is written. If GPT-10 uses utf8 at the input I will eat a shoe.
-
Process Supervision and Error Recovery in RL Training
By
–
Fair but it’s still actively learning to make those errors (just so it can recover from them later), which imo is still a bit weird. In an ideal world you wouldn’t, possibly process supervision is one way to get there even within RL framework. Def agree on “recovery learning”.
-
Hallucinations vs Confabulation: Rethinking RNN Behavior Terminology
By
–
It’s been a decade but yes I believe I hallucinated the term in my 2015 post on unreasonable effectiveness of RNNs. I later became aware that Geoff Hinton used “confabulate”, which is often (but I think not always) a better analogue in human psychology. It’s a bit too specific,
-
Reinforcement Learning Layers in Base Model Training
By
–
I very much hope you continue working on RL! I think it's a misunderstanding that I am suggesting we need some kind of a replacement for RL. That's not accurate and I tried to clear it but did so poorly – they layer. Layer 1 was base model autocomplete.
Layer 2 was instruct -
Collaborating with Grok 5 versus competing against artificial intelligence
By
–
I’d much rather use and collaborate with Grok 5 than compete against it. Though quite similar to chess, and “in the limit” (speaking of physics!), my value add probably trends to ~zero.