Also let's not forget the ability to actually implement your ideas in an efficient, at-scale manner, means you can demonstrate that they work on benchmarks people care about. You'll probably see a lot less interest if you can only prove your brilliant ideas on MNIST. Sometimes
@karpathy
-
Fighting Physics and NVIDIA Compiler Stack for Kernel Optimization
By
–
Great read! My experience is that you’re fighting physics but also the nvidia compiler and the stack overall, and even after pulling *a lot* of tricks we still can’t achieve more than ~80-90% mem bw on many kernels that you’d naively think should be ~100. And the rabbit hole
-
matRBM: Early Open Source Project for Restricted Boltzmann Machines
By
–
It was the worst. But it did have a few really awesome UI features, a built-in debugger, persistent memory, etc. This is one of my first open source projects ever: https://
code.google.com/archive/p/matr
bm/
…
matRBM, a library to train Restricted Boltzmann Machines in Matlab 🙂 Main training loop: -
Evolution of Programming Languages in Machine Learning Research Career
By
–
In roughly all of my experience (Geoff/Ruslan RBM work at UofT, Nando lab at UBC, Andrew Ng lab at Stanford, my 2011 Google internship in baby Google Brain, and ~all computer vision work I was familiar with) it was all only Matlab. I’ve never used Theano but I used Torch in
-
CUDA and C++ Origins Behind AlexNet’s Deep Learning Revolution
By
–
# CUDA/C++ origins of Deep Learning Fun fact many people might have heard about the ImageNet / AlexNet moment of 2012, and the deep learning revolution it started. https://
en.wikipedia.org/wiki/AlexNet What's maybe a bit less known is that the code backing this winning submission to the -
GPT-2 Training Code vs PyTorch’s Universal Capability
By
–
It's worth noting that this code specifically trains GPT-2.
PyTorch trains anything under the sun. -
Fused Classifier Kernel: Algorithmic Improvement Beyond Torch Compile
By
–
this is exactly what we're doing in the fused classifier kernel, and this is an *algorithmic* improvement on top of today's torch compile, which doesn't do this
-
GPU and Disk Constraints: Need for Smaller Dataset Variants
By
–
I'm not only GPU poor but disk poor too. 350GB?
(And ofc doing so wouldn't be representative of the full data distribution)
Also while replying, ideally there could be a "dataset miniseries", e.g. 1B, 10B, 100B, and then full. I think would be very helpful and bandwidth saving. -
cuBLASLt and cuDNN Dependencies for Optimized Performance
By
–
Yes, cuBLASLt for gemms, cuDNN for flash attention
The fp32 version will become more educational and will delete these dependencies. The "mainline" version we just want to be really fast, so we're less discriminating. cuBLASLt I think is ~ok dep, but cuDNN turned out surprisingly -
llm.c Day 24: Multi-GPU Training in C/CUDA Outperforms PyTorch
By
–
Day 24 of llm.c: we now do multi-GPU training, in bfloat16, with flash attention, directly in ~3000 lines of C/CUDA, and it is FAST! We're running ~7% faster than PyTorch nightly, with no asterisks, i.e. this baseline includes all modern & standard bells-and-whistles: mixed