super nice that it just runs on AMD!
have to look into why you seem to be getting less noisy charts, this is on current todo list under getting full determinism.
and yes possibly not exactly comparable to look at electricity costs alone. but i'm not sure there are XTX boxes on
@karpathy
-
AMD Performance and Determinism in ML Training Optimization
By
–
-
Training GPT-3 equivalent models with FineWeb dataset for $500
By
–
GPT-3 model is GPT-2 but trained for longer (300B) tokens and yes on a better dataset. FineWeb is a good dataset, so you can train your own like this. It will cost ~$500. Use -b 32 -t 2048 instead to use the 2048 GPT-3 context length to be accurate.
-
GPT-3 4th Anniversary: Re-training Smallest Model Achievement
By
–
Apparently today is the 4th year anniversary of GPT-3! https://
arxiv.org/abs/2005.14165 Which I am accidentally celebrating by re-training the smallest model in the miniseries right now :). HellaSwag 33.7 (Appendix H) almost reached this a few steps ago (though this is only 45% of the -
AI Capabilities Advancing Faster Than Evaluation Methods Can Track
By
–
The capabilities are improving so fast that evals can't keep up
-
LocalLlama community feedback as crucial evaluation metric
By
–
r/LocalLlama comments section remains a very important evals cross-check no matter what 🙂
-
LLM Evaluation Improvements and the Challenge of Creating Good Benchmarks
By
–
Nice, a serious contender to @lmsysorg in evaluating LLMs has entered the chat. LLM evals are improving, but not so long ago their state was very bleak, with qualitative experience very often disagreeing with quantitative rankings. This is because good evals are very difficult
-
Optimizing batch size and throughput for GPT-3 training
By
–
Okay that's good to know 🙂
I was just following the GPT-3 paper numbers in the table, but like I mentioned it's possible the settings are way too conservative.
Few more things to try: we want to increase the batch size as much as possible to get higher tok/s. Are you using -r 1 -
H100 and FP8 optimizations for performance improvement
By
–
Nice! H100 is a great "free win" to bring this down.
Turning on fp8 for GEMMs would be the other source of really solid improvement, imminently -
Dealing with random MPI hangs in large model training
By
–
I thought I didn't have to deal with these, but already the 350M model (14 hours of 8 GPUs working) sometimes randomly hangs with a cryptic MPI error once in a while. So I have to put the whole optimization into a `while 1` loop and a script that watches the log file and sends
-
Single Node Training vs Large-Scale Multi-GPU Distributed Runs
By
–
But those were also much much bigger runs, so it's a lot more impressive. This was on a single node so you don't need to deal with any cross-node interconnect. It starts to get a lot more fun when you have to keep track of O(10,000) GPUs all at once. For a very specific