fair criticism of the original idea too.
i guess the question is whether such functionality can practically and imminently work, without going full magic (/AGI)
@karpathy
-
Questioning Practical Feasibility of New Functionality Without AGI
By
–
-
LLM as Ultimate Compiler: From Pseudocode to Execution
By
–
Fair 🙂 I love the idea of an LLM as the ultimate compiler. Humans only write something like Python code, but even then it's just the prompt to get the idea across. So it's more like psuedocode and doesn't even have to be valid Python. Actually I already do this once in a while
-
LLM Compile: Converting Python Repositories into Optimized CUDA Code
By
–
torch.compile is cool but LLM compile: takes your .py repo as string and outputs a brand new, custom, from scratch, minimal code repository directly running your network in highly optimized CUDA
-
Deep Learning Performance Optimization: Complexity and Resources
By
–
This post became popular; Few more thoughts / pointers on the topic for the interested reader. Example of the complexity involved: @cHHillee has a great post "Making Deep Learning Go Brrrr From First Principles" https://
horace.io/brrr_intro.html
I was always struck by this diagram from -
GPT-2 Model Now Implemented Directly in Silicon Hardware
By
–
GPT-2 etched directly into silicon let’s go 🙂
-
Learning to Use AI: Understanding Its Strengths and Limitations
By
–
It’s like insisting to walk when you can take a bike. It gets the hard things wrong but all the easy things right, very helpful and much faster. You have to learn what it can and can’t do.
-
GPT-2 as Ideal Class Project for Educational Book
By
–
GPT-2 turns out to be a perfect class project for that book (which I really like and own)
-
Explaining LLM.c: Simplifying Large Language Model Training
By
–
# explaining llm.c in layman terms Training Large Language Models (LLMs), like ChatGPT, involves a large amount of code and complexity. For example, a typical LLM training project might use the PyTorch deep learning library. PyTorch is quite complex because it implements a very https://
t.co/dkpZGRvkmU -
CUDA kernels for GPT-2 forward pass implementation in llm.c
By
–
Okay I did a first quick pass of naive CUDA kernels for the forward pass of GPT-2 and pushed everything to one file in llm.c, Still only ~1000 lines of code: https://
github.com/karpathy/llm.c
/blob/master/train_gpt2.cu
… Current per iteration timings on my Lambda box <3 A100 40GB PCIe, B=4, T=1024:
– llm.c: 111ms
– -
Small Models Utility in Unit Tests and Tiny Applications
By
–
Small models are super useful for unit tests, tiny applications