this is exactly what we're doing in the fused classifier kernel, and this is an *algorithmic* improvement on top of today's torch compile, which doesn't do this
@karpathy
-
GPU and Disk Constraints: Need for Smaller Dataset Variants
By
–
I'm not only GPU poor but disk poor too. 350GB?
(And ofc doing so wouldn't be representative of the full data distribution)
Also while replying, ideally there could be a "dataset miniseries", e.g. 1B, 10B, 100B, and then full. I think would be very helpful and bandwidth saving. -
cuBLASLt and cuDNN Dependencies for Optimized Performance
By
–
Yes, cuBLASLt for gemms, cuDNN for flash attention
The fp32 version will become more educational and will delete these dependencies. The "mainline" version we just want to be really fast, so we're less discriminating. cuBLASLt I think is ~ok dep, but cuDNN turned out surprisingly -
llm.c Day 24: Multi-GPU Training in C/CUDA Outperforms PyTorch
By
–
Day 24 of llm.c: we now do multi-GPU training, in bfloat16, with flash attention, directly in ~3000 lines of C/CUDA, and it is FAST! We're running ~7% faster than PyTorch nightly, with no asterisks, i.e. this baseline includes all modern & standard bells-and-whistles: mixed
-
Interactive portrait with vision, hearing, and recognition capabilities
By
–
The portrait can see and hear you and talk to you just like in the movie, and recognize you as the second factor.
-
Andrej Karpathy wants to buy a living portrait from Hogwarts
By
–
The living portraits at Hogwarts are now technologically quite possible. Would like to buy one and enter my house this way pic.twitter.com/Gqq1VJQl6V
— Andrej Karpathy (@karpathy) 2 mai 2024The living portraits at Hogwarts are now technologically quite possible. Would like to buy one and enter my house this way
-
LLMs in Space: Hardening llm.c to NASA Safety Standards
By
–
Clearly LLMs must one day run in Space Step 1 we harden llm.c to pass the NASA code standards and style guides, certifying that the code is super safe, safe enough to run in Space. https://
en.wikipedia.org/wiki/The_Power
_of_10:_Rules_for_Developing_Safety-Critical_Code
… (see the linked PDF)
LLM training/inference in principle should be super -
Code Simplicity vs Compression: The LOC Paradox
By
–
[gif] me trying to read tinygrad code earlier 😀
— Andrej Karpathy (@karpathy) 25 avril 2024
I think the LOC requirements (which are only a proxy for simplicity) led to too great compression. You wouldn't brag about your .min.js code being 1 LOC. Imo it would be a lot more simple if the code was given room to breathe and… pic.twitter.com/K4krMQdRkU[gif] me trying to read tinygrad code earlier 😀 I think the LOC requirements (which are only a proxy for simplicity) led to too great compression. You wouldn't brag about your .min.js code being 1 LOC. Imo it would be a lot more simple if the code was given room to breathe and
-
Mastering AI coding assistance through practice and proper prompting
By
–
I've also gotten really good at it. I think people who dislike it must have given up too early. You have to learn what to expect, what works, what doesn't work, how to position your cursor (e.g. to not get distracting suggestions), and how to prompt it well via code/comments
-
H100 GPUs as the Ultimate Source of Happiness
By
–
Money can't buy happiness.
Just like an H100.
H100 = happiness.