I love that the prize recognizes foundational work in all 3 pillars of AI (data, compute, algorithms), congratulations Fei-Fei!!
@karpathy
-
PyTorch Framework Transition: Pleasant Developer Experience
By
–
Great run Soumith! Of all the deep learning framework transitions I've been through where I re-wrote ~all of my code (matlab -> caffe -> numpy -> torch -> pytorch), the PyTorch one was most pleasant and now significantly longest lasting. It hit a jackpot of the time in the
-
AGI Definition: Feature-Complete Remote Worker with High Reliability
By
–
When I say AGI I mean a "feature complete" build of a remote worker with a lot of 9s (think: a bit like current Autopilot FSD, maybe plus a few more iterations). This is the original definition I've stuck to forever. Separate from the diffusion /implementation of it across
-
PyTorch MPS Backend Bug: Silent Tensor Contiguity Failures
By
–
Beautiful technical debugging detective longread that starts with a suspicious loss curve and ends all the way in the Objective-C++ depths of PyTorch MPS backend of addcmul_ that silently fails on non-contiguous output tensors. I wonder how long before an LLM can do all of this.
-
AI Model Monetization and User Empowerment Strategy
By
–
It’s my model I can do whatever I want. So now I am king. You can be too for $100.
-
Building Free AI: NanoChat’s Vision Beyond Traditional Assistants
By
–
I find it surprisingly addicting. I have so many ideas for personality, style, capabilities. I want nanochat to be a free AI, like Dobby the free elf, not some "assistant".
-
Teaching AI Models New Skills: The R-Counting Example
By
–
Last night I taught nanochat d32 how to count 'r' in strawberry (or similar variations). I thought this would be a good/fun example of how to add capabilities to nanochat and I wrote up a full guide here: https://
github.com/karpathy/nanoc
hat/discussions/164
… This is done via a new synthetic task -
Frontier LLM Production Stack Innovation Discussion
By
–
Ty, I just meant it hasn’t (to my knowledge) created a whole new stage in the production stack of a frontier LLM, or had a contribution on that level.
-
NanoChat Live Tokenization During Training Optimization
By
–
atm nanochat does not pretokenize, instead tokenization is done live during training in the dataloader, which is ok and gives the CPU something to do while the GPUs are working. So I didn't try too hard to optimize this part.
-
Regex bottleneck in tokenizer requires expert optimization
By
–
Yep, the use of regex is both a huge dependency and huge bottleneck in the tokenizer. I think it's a beautiful project to try to do this correctly, but I'd need someone who is really familiar with regex to pitch in and also a large test suite to make sure. I'd love to merge such