The last few iters you may be seeing early signs of instability. I saw the same at around 250B tokens, it slowly gets worse and worse and then loss spikes. I haven’t stabilized it yet, right now seeing how easy it goes away with simple solutions, resetting the data loader etc
@karpathy
-
Tripling Learning Rate: Unexpected Optimization Discovery
By
–
It’s interesting that you can 3X the LR. You’d expect the original paper to be well tuned near what is tolerable.
-
GPT-3 Models Will Soon Be Surpassed by Larger Models
By
–
GPT3-124M. But even the 175B will fall not too far from now
-
Andrej Karpathy’s Favorite Suno AI-Generated Music Creations
By
–
Can I just say I loooove Suno. Some of my favorites: Dog dog dog dog dog dog dog dog woof woof https://
suno.com/song/1783c864-
18fb-440f-bc51-15701a19e4b5
…
Chemical elements https://
suno.com/song/5f324463-
08a7-490e-b9c5-f8e2d399baa9
…
train_gpt2.c header (who did this lol) https://
suno.com/song/2a210337-
62fc-49f8-8850-9af12e06e6e0
…
Suno tutorial (in Suno!): https://
suno.com/song/d960e84a-
1b03-46a2-999e-2a896a56bd57
… Many -
AMD Performance and Determinism in ML Training Optimization
By
–
super nice that it just runs on AMD!
have to look into why you seem to be getting less noisy charts, this is on current todo list under getting full determinism.
and yes possibly not exactly comparable to look at electricity costs alone. but i'm not sure there are XTX boxes on -
Training GPT-3 equivalent models with FineWeb dataset for $500
By
–
GPT-3 model is GPT-2 but trained for longer (300B) tokens and yes on a better dataset. FineWeb is a good dataset, so you can train your own like this. It will cost ~$500. Use -b 32 -t 2048 instead to use the 2048 GPT-3 context length to be accurate.
-
GPT-3 4th Anniversary: Re-training Smallest Model Achievement
By
–
Apparently today is the 4th year anniversary of GPT-3! https://
arxiv.org/abs/2005.14165 Which I am accidentally celebrating by re-training the smallest model in the miniseries right now :). HellaSwag 33.7 (Appendix H) almost reached this a few steps ago (though this is only 45% of the -
AI Capabilities Advancing Faster Than Evaluation Methods Can Track
By
–
The capabilities are improving so fast that evals can't keep up
-
LocalLlama community feedback as crucial evaluation metric
By
–
r/LocalLlama comments section remains a very important evals cross-check no matter what 🙂
-
LLM Evaluation Improvements and the Challenge of Creating Good Benchmarks
By
–
Nice, a serious contender to @lmsysorg in evaluating LLMs has entered the chat. LLM evals are improving, but not so long ago their state was very bleak, with qualitative experience very often disagreeing with quantitative rankings. This is because good evals are very difficult