GPT-2 etched directly into silicon let’s go 🙂
@karpathy
-
Learning to Use AI: Understanding Its Strengths and Limitations
By
–
It’s like insisting to walk when you can take a bike. It gets the hard things wrong but all the easy things right, very helpful and much faster. You have to learn what it can and can’t do.
-
GPT-2 as Ideal Class Project for Educational Book
By
–
GPT-2 turns out to be a perfect class project for that book (which I really like and own)
-
Explaining LLM.c: Simplifying Large Language Model Training
By
–
# explaining llm.c in layman terms Training Large Language Models (LLMs), like ChatGPT, involves a large amount of code and complexity. For example, a typical LLM training project might use the PyTorch deep learning library. PyTorch is quite complex because it implements a very https://
t.co/dkpZGRvkmU -
CUDA kernels for GPT-2 forward pass implementation in llm.c
By
–
Okay I did a first quick pass of naive CUDA kernels for the forward pass of GPT-2 and pushed everything to one file in llm.c, Still only ~1000 lines of code: https://
github.com/karpathy/llm.c
/blob/master/train_gpt2.cu
… Current per iteration timings on my Lambda box <3 A100 40GB PCIe, B=4, T=1024:
– llm.c: 111ms
– -
Small Models Utility in Unit Tests and Tiny Applications
By
–
Small models are super useful for unit tests, tiny applications
-
Model Size Discussion: 100M and 1B Parameters Questioned
By
–
How small? Probably not small enough 100M and 1B models
-
LLM Agent Solutions and Training Data Contamination Concerns
By
–
my biggest concern is that an LLM Agent can solve it in 1-2 years but by then related forks and code and discussions seep into its training data in an undefined way, making the result unsatisfying.
-
LLM Agents Challenge: Porting PyTorch GPT-2 Training to C
By
–
Btw writing the llm.c training code would imo be a very interesting, impressive, self-contained and very meta challenge for LLM agents. The prompt is: Take the PyTorch code train_gpt2.py
And write, compile and unit test a single .c file that reproduces the training: train_gpt2.c -
PyTorch Layers Migration to C: Tutorial and Implementation Guide
By
–
I added a quick crappy tutorial on how PyTorch layers are moved to C, with a few possibly helpful pointers: https://
github.com/karpathy/llm.c
/blob/master/doc/layernorm/layernorm.md
…