Also reminded of this blog post from ~12 years ago. I classified CIFAR10 manually and got… 94%! SOTA then was ~80%, certainly not in 10 seconds. Then I predicted we'd top out around 85-90% (lol). 12 years later: 94% is 10 seconds with one 600-line script https://
karpathy.github.io/2011/04/27/man
ually-classifying-cifar10/
…
@karpathy
-
CIFAR-10 Classification Progress: From Manual 94% to Automated Scripts
By
–
-
CIFAR10 Training to 94% in Under 10 Seconds on A100
By
–
More on cramming: CIFAR10 hyperlightspeedbench.
Train CIFAR10 to 94% in under 10 seconds on a single A100. With a single readable 600-line http://
main.py, bunch of nice tricks implemented within. -
Deep Learning Performance: Empirical Nature and High Experimental Throughput Necessity
By
–
A good display of how empirical and setting-dependent deep learning can still be, and what driving up performance looks like. In any setting it's not so much "here's how you can improve" but "here's the 10 things you should try". And why high experimental throughput is necessary.
-
Appreciation for Cramming Paper’s Experimental Approach and Code Release
By
–
(finally got around to reading in full). Amusing to read so many negative result attempts back to back to incorporate previous papers/ideas (at least in the cramming setting). Like the inline experimental result style. Like the nice code release. Like the "cramming" benchmark.
-
Building a Transformer with simplified layers in nanoGPT
By
–
Yep, that's the one! (as @Thom_Wolf linked earlier too). I'd expect it's possible to build a Transformer with that kind of layer alone, would look much more pleasing. Will see if I can prototype in nanoGPT.
-
Simplified Transformer Architecture with Unified Block Design
By
–
TLDR: A much simpler Transformer with a single type of block wired up to a residual pathway in both parallel and in series is possible but to my knowledge has not yet been convincingly demonstrated. Bit more detail @ https://
github.com/karpathy/rando
mfun/blob/master/transformer_unify.ipynb
… -
Transformer Block Unification: MLP and Attention Similarity
By
–
Random quick note on Transformer block unification. People are usually a bit surprised that the MLP and Attention blocks that repeat in a Transformer can be re-formated to look very similar, likely unifiable. The MLP block just attends over data-independent {key: value} nodes:
-
Batchnorm as a Challenging Interview Question for Candidates
By
–
For a while I thought an excellent interview question would be to ask the candidate about batchnorm and measure the amount to which their face contorts
-
GPT as Backend: LLM as English Interpreter for Development
By
–
"GPT is all you need for backend". This was the most inspirational project from the hackathon over the weekend, hard to stop thinking about. LLM is a kind of equivalent of the Python interpreter, except it interprets English, and has knowledge and common sense.
-
English Becomes the Hottest New Programming Language
By
–
The hottest new programming language is English