5/ We thus have (at least) 3 serious codebases for evaluating on the same MMLU dataset:
– "Original implementation" from the MMLU benchmark authors
– "HELM implementation" from Stanford
– "Harness implementation" from EleutherAI (recently updated – see the end of the thread)
@thom_wolf
-
Three Major MMLU Evaluation Codebases Compared
By
–
-
LLaMA Team Uses Original MMLU Benchmark Evaluation Code
By
–
3/ It turns out the LLaMA team used the original evaluation code proposed by the authors of the MMLU benchmark (find it at https://
github.com/hendrycks/test) Let's call it the "original implementation" -
Open LLM Leaderboard: Evaluation Framework and Results
By
–
1/ First, what's the Open LLM Leaderboard? Just a wrapper running the @AiEleuther evaluation harness https://
github.com/EleutherAI/lm-
evaluation-harness
… on the Hugging Face cluster spare cycles + storing/displaying results at https://
huggingface.co/spaces/Hugging
FaceH4/open_llm_leaderboard
… -
LLaMA 65B MMLU Benchmark Discrepancy Analysis
By
–
2/ For one evaluation, MMLU (
https://
arxiv.org/abs/2009.03300), the community was surprised that the leaderboard numbers for the top model, LLaMA 65B, were significantly lower than the numbers in the published LLaMa paper: a 30% difference! We dived in a rabbit hole to understand -
Open LLM Leaderboard evaluation accuracy analysis findings
By
–
A on "what was going on with the Open LLM Leaderboard?" its numbers didn't match the ones reported in LLaMA paper so we dived in it and wrote a blog post of learnings! Here's the thread version for those of you who didn't want to read a blog post
-
Prompt Changes Show Minimal Impact on Ranking Results
By
–
All prompts changes had very little effect in our experiments, in particular on ranking. Probably should add more details on this.
-
Open LLM Leaderboard Evaluation Discrepancies and Methodology
By
–
What was going on with the Open LLM Leaderboard? Its numbers didn't match the ones reported in the LLaMA paper! We've decided to dive in this rabbit hole with friends from the LLaMA & Falcon teams and got back with a blog post of learnings & surprises: https://
huggingface.co/blog/evaluatin
g-mmlu-leaderboard
… -
MosaicML Releases New 30B Open Source Language Model
By
–
The just release 30B model from MosaicML looks really great! Nice (big) size, OSS apache-2 licence and long context! take a look at the thread for more details
-
OpenAI’s Rapid Execution in AI Model and Product Shipping
By
–
moats or no-moats on gpt4 it's still super impressive how OpenAI is shipping new models, products and ideas so fast and consistently. amazing execution
-
High Quality Data Distillation from Large Noisy Corpora
By
–
More the reverse I guess: current trend is to distill high quality data from large (noisy) corpora