but even these evals are already fairly specific, would be interesting to see a broader eval coverage.
@karpathy
-
Exporting Model Weights for Faster Evaluation Infrastructure
By
–
Instead of building them out inside llm.c it might be faster to export the model weights into "common infra" and run evals with that. I don't have time to get around to it right away but made an Issue a few days ago for someone to potentially take a look.
-
Pretraining Datasets and Evaluation Metrics for Multilingual Models
By
–
Yes, definitely, my last tweet few seconds ago is also on this point. And many pretraining datasets also care about e.g. multilignual, code, math, etc., so it's not clear how those evals would be affected.
-
GPT-3 Training Surpasses Expected Performance on FineWeb Dataset
By
–
Example here is the llm.c GPT-3 (124M) training on FineWeb (figure cropped at 250B tokens), we seem to surpass GPT-3 HellaSwag (green line) at ~150B tokens, per paper expected this to be at 300B tokens. Will re-run with FineWeb-Edu. I do want to be a bit careful on conclusions
-
llm.c Outperforming GPT-2/3 with Fewer Training Tokens
By
–
In llm.c pretraining we were already mildly perplexed why seem to be outperforming GPT-2 & 3 (124M) training on just 10B tokens instead of something closer to 100-300B, per the original papers. I suspect a good chunk of it may be just the dataset quality, so I'm eager to retrain
-
FineWeb-Edu: High-Quality LLM Dataset Filtering for Better Learning
By
–
Awesome and highly useful: FineWeb-Edu High quality LLM dataset filtering the original 15 trillion FineWeb tokens to 1.3 trillion of the highest (educational) quality, as judged by a Llama 3 70B. +A highly detailed paper. Turns out that LLMs learn a lot better and faster
-
Training instability detected around 250B tokens during model training
By
–
The last few iters you may be seeing early signs of instability. I saw the same at around 250B tokens, it slowly gets worse and worse and then loss spikes. I haven’t stabilized it yet, right now seeing how easy it goes away with simple solutions, resetting the data loader etc
-
Tripling Learning Rate: Unexpected Optimization Discovery
By
–
It’s interesting that you can 3X the LR. You’d expect the original paper to be well tuned near what is tolerable.
-
GPT-3 Models Will Soon Be Surpassed by Larger Models
By
–
GPT3-124M. But even the 175B will fall not too far from now
-
Andrej Karpathy’s Favorite Suno AI-Generated Music Creations
By
–
Can I just say I loooove Suno. Some of my favorites: Dog dog dog dog dog dog dog dog woof woof https://
suno.com/song/1783c864-
18fb-440f-bc51-15701a19e4b5
…
Chemical elements https://
suno.com/song/5f324463-
08a7-490e-b9c5-f8e2d399baa9
…
train_gpt2.c header (who did this lol) https://
suno.com/song/2a210337-
62fc-49f8-8850-9af12e06e6e0
…
Suno tutorial (in Suno!): https://
suno.com/song/d960e84a-
1b03-46a2-999e-2a896a56bd57
… Many