What is larger or higher here, do you have example annealing schedule you’ve found worked well?
@karpathy
-
Bug Fix: Gradient Norm Clipping Synchronization Issue with ZeRO-1
By
–
There was a bug with gradient norm clipping, it was incorrectly synchronized across GPUs when using ZeRO-1, which may have contributed to the loss spike I saw earlier. It's fixed now on master. More generally as of yesterday though we've re-established full and exact equality to
-
Question about Flash Attention in cuDNN Implementation
By
–
Sorry can you expand? Are you saying this might be due to the flash attention inside cuDNN?
-
Broader evaluation coverage needed for AI models
By
–
but even these evals are already fairly specific, would be interesting to see a broader eval coverage.
-
Exporting Model Weights for Faster Evaluation Infrastructure
By
–
Instead of building them out inside llm.c it might be faster to export the model weights into "common infra" and run evals with that. I don't have time to get around to it right away but made an Issue a few days ago for someone to potentially take a look.
-
Pretraining Datasets and Evaluation Metrics for Multilingual Models
By
–
Yes, definitely, my last tweet few seconds ago is also on this point. And many pretraining datasets also care about e.g. multilignual, code, math, etc., so it's not clear how those evals would be affected.
-
GPT-3 Training Surpasses Expected Performance on FineWeb Dataset
By
–
Example here is the llm.c GPT-3 (124M) training on FineWeb (figure cropped at 250B tokens), we seem to surpass GPT-3 HellaSwag (green line) at ~150B tokens, per paper expected this to be at 300B tokens. Will re-run with FineWeb-Edu. I do want to be a bit careful on conclusions
-
llm.c Outperforming GPT-2/3 with Fewer Training Tokens
By
–
In llm.c pretraining we were already mildly perplexed why seem to be outperforming GPT-2 & 3 (124M) training on just 10B tokens instead of something closer to 100-300B, per the original papers. I suspect a good chunk of it may be just the dataset quality, so I'm eager to retrain
-
FineWeb-Edu: High-Quality LLM Dataset Filtering for Better Learning
By
–
Awesome and highly useful: FineWeb-Edu High quality LLM dataset filtering the original 15 trillion FineWeb tokens to 1.3 trillion of the highest (educational) quality, as judged by a Llama 3 70B. +A highly detailed paper. Turns out that LLMs learn a lot better and faster