A lot of fun in the Appendix, e.g. how GeLU can be used for multiplication / bypassing it as identity, use of LayerNorm for division, or bypassing that as identity, etc.
@karpathy
-
Transformer Equations as Low-Level Microcode and Assembly Instructions
By
–
Nice! Like the track of work. Equations of Transformer are a bit like low-level microcode, this track tries to "go up" to uncover an implied assembly instruction set (e.g. RAW "read-arithmetic-write" operator?), and implemented algorithms on top of that for e.g. ridge regression.
-
New Neural Network Zero to Hero YouTube Series in Development
By
–
(I think he means my new NN: Zero to Hero series https://
youtube.com/playlist?list=
PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ
… , which I'm still building out) -
Comparing Stable Diffusion 1.5 vs 2.0 Image Generation Quality
By
–
easy to compare a lot of images from both models on http://
stableboost.ai , e.g. "cute dog cooking tacos, photorrealistic", grid of boosted images from 1.5 (left) and 2.0 (right). 2.0 looking more distorted, cartoony, simpler, ignores text more. may need more prompt engineering -
Stable Diffusion 2.0 Shows Quality Decline Compared to 1.5
By
–
plot twist: stable diffusion 2.0 looks quite a bit worse on the few prompts i've tried so far compared to 1.5 (even not including celebrities/artists). Running theory seems to be this is due to an aggressive data sanitization campaign since the original release (?).
-
AI Performance Limitations on Unconstrained Web Search Tasks
By
–
It works well when it’s force constrained to sites like reddit twitter etc. it just can’t be trusted to find good sites
-
General-Purpose Neural Networks Through Simple Scalable Objectives
By
–
when the core unlock was achieving a kind of general-purpose computer neural net via simple scalable objectives that have strong training signal (many bits of contraints per training example). Like language modeling, and not like reinforcement learning.
So that was interesting 😀 -
Mispredicting scaling potential: Early distraction with reinforcement learning
By
–
But I still mispredicted in how much fertile ground there was in scaling up the paradigm. Like many others in AI I got distracted by Reinforcement Learning too soon, a kind of putting the cart before the horse, …
-
A Decade of AI Progress: Language Models and Neural Networks
By
–
I wrote this thread because I spent the last ~decade, obsessing over directions that would make fastest progress in AI, and was very interested in language models (e.g. my semi-famous 2015 post "The Unreasonable Effectiveness of Recurrent Neural Networks" https://
karpathy.github.io/2015/05/21/rnn
-effectiveness/
…) -
Scaling Language Models: Neural Networks as General-Purpose Text Computers
By
–
TLDR: LMs have been around forever. Not obvious finding: turns out that if you scale up the training set and use a powerful enough neural net (Transformer), the network becomes a kind of general-purpose computer over text.