Some personal news: I am joining OpenAI (again :)). Like many others both in/out of AI, I am very inspired by the impact of their work and I have personally benefited greatly from it. The future potential is especially exciting; it is a great pleasure to jump back in and build!
@karpathy
-
Single Flag Changes Can Match Months of Optimization Work
By
–
I love how sometimes changing one integer/flag can have the same impact as a 1 month optimization project. You just know there is some OMP_NEVER_HEARD_OF=3 that gets addition 3% MFU. Or my personal favorite – that undocumented bios flag that only 4 people on Earth know exists 😀
-
Linear and LayerNorm Biases Appear Useless in Experiments
By
–
I'll give it a shot! Btw it is biases in both Linear and LayerNorm that appear to be useless (from my admittedly smaller scale experiments).
-
Vocabulary Size Optimization: 25% Speed Boost Through Kernel Alignment
By
–
The most dramatic optimization to nanoGPT so far (~25% speedup) is to simply increase vocab size from 50257 to 50304 (nearest multiple of 64). This calculates added useless dimensions but goes down a different kernel path with much higher occupancy. Careful with your Powers of 2.
-
CIFAR-10 Classification Progress: From Manual 94% to Automated Scripts
By
–
Also reminded of this blog post from ~12 years ago. I classified CIFAR10 manually and got… 94%! SOTA then was ~80%, certainly not in 10 seconds. Then I predicted we'd top out around 85-90% (lol). 12 years later: 94% is 10 seconds with one 600-line script https://
karpathy.github.io/2011/04/27/man
ually-classifying-cifar10/
… -
CIFAR10 Training to 94% in Under 10 Seconds on A100
By
–
More on cramming: CIFAR10 hyperlightspeedbench.
Train CIFAR10 to 94% in under 10 seconds on a single A100. With a single readable 600-line http://
main.py, bunch of nice tricks implemented within. -
Deep Learning Performance: Empirical Nature and High Experimental Throughput Necessity
By
–
A good display of how empirical and setting-dependent deep learning can still be, and what driving up performance looks like. In any setting it's not so much "here's how you can improve" but "here's the 10 things you should try". And why high experimental throughput is necessary.
-
Appreciation for Cramming Paper’s Experimental Approach and Code Release
By
–
(finally got around to reading in full). Amusing to read so many negative result attempts back to back to incorporate previous papers/ideas (at least in the cramming setting). Like the inline experimental result style. Like the nice code release. Like the "cramming" benchmark.
-
Building a Transformer with simplified layers in nanoGPT
By
–
Yep, that's the one! (as @Thom_Wolf linked earlier too). I'd expect it's possible to build a Transformer with that kind of layer alone, would look much more pleasing. Will see if I can prototype in nanoGPT.
-
Simplified Transformer Architecture with Unified Block Design
By
–
TLDR: A much simpler Transformer with a single type of block wired up to a residual pathway in both parallel and in series is possible but to my knowledge has not yet been convincingly demonstrated. Bit more detail @ https://
github.com/karpathy/rando
mfun/blob/master/transformer_unify.ipynb
…