The history of computing is repeating in an echo, except replace computers that do precise arithmetic on bytes with computers that do statistical arithmetic on tokens.
@karpathy
-
Scheduling Computational Workloads to Run on Humans
By
–
# scheduling workloads to run on humans Some computational workloads in human organizations are best "run on a CPU": take one single, highly competent person and assign them a task to complete in a single-threaded fashion, without synchronization. Usually the best fit when
-
PyTorch Performance Improvements with Nightly Build and Optimizations
By
–
THE REVENGE OF PYTORCH
just kidding 🙂 @cHHillee (from PyTorch team) was kindly able to help improve the PyTorch baseline, done by 1) upgrading to nightly, 2) using the "compound" F.sdpa (scaled dot product attention) layer directly, and turning on a torch compile flag: -
llm.c Optimization: Matching PyTorch Performance After Bug Fix
By
–
Highly amusing update, ~18 hours later: llm.c is now down to 26.2ms/iteration, exactly matching PyTorch (tf32 forward pass). We discovered a bug where we incorrectly called cuBLAS in fp32 mathmode . And ademeure contributed a more optimized softmax kernel for very long rows
-
Apple Silicon GPU Programming Requires Different Libraries and Approach
By
–
Apple silicon is a whole different thing but yes it also has a GPU, and the code can also be targeted to it in a similar way, but all of the libraries and details change (Metal instead of CUDA) so it’s a whole separate effort.
-
llm.c Performance Optimization: Closing Gap with PyTorch
By
–
A few new CUDA hacker friends joined the effort and now llm.c is only 2X slower than PyTorch (fp32, forward pass) compared to 4 days ago, when it was at 4.2X slower The biggest improvements were:
– turn on TF32 (NVIDIA TensorFLoat-32) instead of FP32 for matmuls. This is a -
Questioning Practical Feasibility of New Functionality Without AGI
By
–
fair criticism of the original idea too.
i guess the question is whether such functionality can practically and imminently work, without going full magic (/AGI) -
LLM as Ultimate Compiler: From Pseudocode to Execution
By
–
Fair 🙂 I love the idea of an LLM as the ultimate compiler. Humans only write something like Python code, but even then it's just the prompt to get the idea across. So it's more like psuedocode and doesn't even have to be valid Python. Actually I already do this once in a while
-
LLM Compile: Converting Python Repositories into Optimized CUDA Code
By
–
torch.compile is cool but LLM compile: takes your .py repo as string and outputs a brand new, custom, from scratch, minimal code repository directly running your network in highly optimized CUDA
-
Deep Learning Performance Optimization: Complexity and Resources
By
–
This post became popular; Few more thoughts / pointers on the topic for the interested reader. Example of the complexity involved: @cHHillee has a great post "Making Deep Learning Go Brrrr From First Principles" https://
horace.io/brrr_intro.html
I was always struck by this diagram from