Consider being a labeler for an LLM. The prompt is “give me a random number between 1 and 10”. What SFT & RM labels do you contribute? What does this do the network when trained on? In subtle way this problem is present in every prompt that does not have a single unique answer.
@karpathy
-
Five Years Between Transformer Attention and FlashAttention Innovation
By
–
"5 years between Self-Attention Is All You Need and FlashAttention"
quite incredible stat, gives a pause -
Modeling Different Coprocessors with Varying Instruction Sets
By
–
another layer of the onion 🙂 maybe can you model it like having different kinds of coprocessors with different instruction sets / reliability rates
-
Visualizing Computation DAGs: Identifying Narrow and Wide Bottlenecks
By
–
sorry i just made that word up in my head right now, didn't mean to hijack some existing term. I meant – imagine the computation as a DAG, lay it out in your head, and look for "narrow" and "wide" parts. Open to alternatives for future use!
-
Computing History Repeats: From Bytes to Tokens
By
–
The history of computing is repeating in an echo, except replace computers that do precise arithmetic on bytes with computers that do statistical arithmetic on tokens.
-
Scheduling Computational Workloads to Run on Humans
By
–
# scheduling workloads to run on humans Some computational workloads in human organizations are best "run on a CPU": take one single, highly competent person and assign them a task to complete in a single-threaded fashion, without synchronization. Usually the best fit when
-
PyTorch Performance Improvements with Nightly Build and Optimizations
By
–
THE REVENGE OF PYTORCH
just kidding 🙂 @cHHillee (from PyTorch team) was kindly able to help improve the PyTorch baseline, done by 1) upgrading to nightly, 2) using the "compound" F.sdpa (scaled dot product attention) layer directly, and turning on a torch compile flag: -
llm.c Optimization: Matching PyTorch Performance After Bug Fix
By
–
Highly amusing update, ~18 hours later: llm.c is now down to 26.2ms/iteration, exactly matching PyTorch (tf32 forward pass). We discovered a bug where we incorrectly called cuBLAS in fp32 mathmode . And ademeure contributed a more optimized softmax kernel for very long rows
-
Apple Silicon GPU Programming Requires Different Libraries and Approach
By
–
Apple silicon is a whole different thing but yes it also has a GPU, and the code can also be targeted to it in a similar way, but all of the libraries and details change (Metal instead of CUDA) so it’s a whole separate effort.
-
llm.c Performance Optimization: Closing Gap with PyTorch
By
–
A few new CUDA hacker friends joined the effort and now llm.c is only 2X slower than PyTorch (fp32, forward pass) compared to 4 days ago, when it was at 4.2X slower The biggest improvements were:
– turn on TF32 (NVIDIA TensorFLoat-32) instead of FP32 for matmuls. This is a