Okay that's good to know 🙂
I was just following the GPT-3 paper numbers in the table, but like I mentioned it's possible the settings are way too conservative.
Few more things to try: we want to increase the batch size as much as possible to get higher tok/s. Are you using -r 1
@karpathy
-
Optimizing batch size and throughput for GPT-3 training
By
–
-
H100 and FP8 optimizations for performance improvement
By
–
Nice! H100 is a great "free win" to bring this down.
Turning on fp8 for GEMMs would be the other source of really solid improvement, imminently -
Dealing with random MPI hangs in large model training
By
–
I thought I didn't have to deal with these, but already the 350M model (14 hours of 8 GPUs working) sometimes randomly hangs with a cryptic MPI error once in a while. So I have to put the whole optimization into a `while 1` loop and a script that watches the log file and sends
-
Single Node Training vs Large-Scale Multi-GPU Distributed Runs
By
–
But those were also much much bigger runs, so it's a lot more impressive. This was on a single node so you don't need to deal with any cross-node interconnect. It starts to get a lot more fun when you have to keep track of O(10,000) GPUs all at once. For a very specific
-
Why 10B Tokens Suffice for GPT Training Performance
By
–
Great question yes I was surprised that 10B seemed enough. I believe GPT-2 was trained on somewhere ~100B tokens. The reason we reach this performance in 10B tokens I think may be the following: 1. FineWeb could just be higher quality than WebText, on a per-token basis. This was
-
Start Latency Issues with Dataset Tokenization in Python
By
–
TIL, will look into! The thing that makes this a bit complicated right now is the start latency. What bloats up the setup time right now is the dataset and its tokenization, which is all done in Python right now. Installing huggingface datasets, downloading FineWeb 10B and
-
H100 GPU Shortage Challenges Training Cost Reduction
By
–
Yep, moving to H100 is one easy way to bring down the estimates here. Sadly I can't find any H100 GPUs
-
Training Loss Noise and Validation Loss Smoothing Explained
By
–
Training loss is evaluated over the batch, i.e. 0.5M tokens. It's noisy but this is expected, you could be iterating through easy or hard documents in the training data. The validation loss is averaged over 20 batches of 0.5M tokens (this is a hyperparameter), so it is smoother.
-
Conservative hyperparameters with potential for 10X speedup
By
–
Agree!! I'm using very conservative settings for a lot of the hyperparameters (following GPT-3 paper when possible) and haven't tried to speed this up at all yet, but I expect a 10X multiplier here should be possible.
-
Reproduce GPT-2 124M in llm.c for $20 in 90 Minutes
By
–
# Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 The GPT-2 (124M) is the smallest model in the GPT-2 series released by OpenAI in 2019, and is actually quite accessible today, even for the GPU poor. For example, with llm.c you can now reproduce this model on one 8X