solid programming, familiarity (/willingness to learn) tensor processing (numpy or torch tensor), small few concepts from basic math and statistics (e.g. function gradient, gaussian distribution, etc.). I'll list this out on the page, ty.
@karpathy
-
Comparing Modern GPT Models to OpenAI’s Early GPT-2 Capabilities
By
–
It’s about as good as OpenAI’s baby GPT-2 from ~4 years ago. (Their paper at that time had models from 124M to 1.3B). Today’s bleeding edge GPTs reach scale (in model size and data size) that requires significant infrastructure and further finetuning to align them (RLHF etc).
-
Evolution from search engines to answer engines and beyond
By
–
Tired: search engine
Wired: answer engine
Inspired: ???
🙂 -
LLMs Need External Tools Like Humans Need Interaction
By
–
LLMs are like a person doing everything just in their head. People wouldn’t get very far like that alone. LLMs wouldn’t either.
-
Clarification on GPT-2 Model Size: 124M vs 1.3B Parameters
By
–
Careful this is the 124M model. The biggest GPT-2 was 1.3B
-
Neural Networks: Zero to Hero Series – Building Networks from Scratch
By
–
(This will be part of my ongoing series Neural Networks: Zero to Hero https://
karpathy.ai/zero-to-hero.h
tml
… , on building neural networks, from scratch, in code. I have tweeted some of these videos individually already) -
Future Plans for GPT-2 Implementation and Educational Content
By
–
I'd like to continue to make it faster, reproduce the other GPT-2 models, then scale up pre-training to bigger models/datasets, then improve the docs for finetuning (the practical use case). Also working on video lecture where I will build it from scratch, hoping out in ~2 weeks.
-
GPT-2 Pre-training: Hardware Requirements and Token Processing Estimates
By
–
Rough example, a decent GPT-2 (124M) pre-training reproduction would be 1 node of 8x A100 40GB for 32 hours, processing 8 GPU * 16 batch size * 1024 block size * 500K iters = ~65B tokens. I suspect this wall clock can still be improved ~2-3X+ without getting too exotic.
-
nanoGPT: Simplest Repository for Training Medium-Sized GPTs
By
–
Didn't tweet nanoGPT yet (quietly getting it to good shape) but it's trending on HN so here it is 🙂 : https://
github.com/karpathy/nanoG
PT
…
Aspires to be simplest, fastest repo for training/finetuning medium-sized GPTs. So far confirmed it reproduced GPT-2 (124M). 2 simple files of ~300 lines -
Long-term work on arxiv-sanity project iterations
By
–
for sure! would love to know a bit more under the hood. I've working on this problem for a _long_ time, arxiv-sanity versions 1,2,3,4,5 and all 😀