23/ …not at all comparable even if they're both called MMLU & evaluated on same dataset Takeaway? Evaluations are strongly tied to implementations–down to minute details. A mere indication of "MMLU score" gives almost no information about how you can compare these numbers
MACHINE LEARNING
-
LLaMA 65B Evaluation Discrepancies Between Implementations
By
–
22/ Say you've trained a perfect LLaMA 65B reproduction & evaluated it with EAI harness (score 0.488). Comparing it to the published number (evaluated w. original implementation, score 0.637), it's a 30% difference so you're likely thinking "Oh no " But these numbers are…
-
Model Evaluation Methods Significantly Impact Scores and Rankings
By
–
21/ Now let’s compare the model scores on these three possible ways to evaluate the models We can see that for the same dataset, both absolute scores and model rankings are very sensitive to the evaluation method we decide to use.
-
Probability-based sequence generation normalization in language models
By
–
19/ In this case, we use probabilities again but this time we compute the probability of generating the full answer sequence, not just the letter: we sum the log of the probabilities and compute a normalization by dividing by the number of tokens to not penalize longer sequences.
-
Harness Evaluation Framework for Large Language Models
By
–
18/ Harness: Now we finally turn to the EleutherAI Harness implementation (as of January 2023) which was used to compute the numbers for the Open LLM Leaderboard. Here is yet another way to compute a score for the model on the very same evaluation dataset! Let's take a look:
-
Model Evaluation: Uncounted Prediction Errors in Multiple Choice
By
–
14/ As you can see, we compare the probabilities predicted by the model, on the four answers *only*. But sometimes that model would have made a mistake (generating "Zygote" here) which is not counted as a mistake…
-
Two main ways to evaluate model outputs
By
–
11/ In summary, we have two main ways to evaluate a model outputs:
1. Get the probabilities that some tokens are continuations of the prompt => compare these probabilities together
2. Get the text generation from the model => compare these text generations to possible choices -
HELM Benchmark Implementation Analysis for MMLU Evaluation
By
–
4/ Diving further, we found yet another serious implementation for evaluating on the very same MMLU dataset: the code used in the HELM benchmark https://
crfm.stanford.edu from @StanfordCRFM
: https://
github.com/stanford-crfm/
helm
… Let's call it the "HELM implementation" -
Three Major MMLU Evaluation Codebases Compared
By
–
5/ We thus have (at least) 3 serious codebases for evaluating on the same MMLU dataset:
– "Original implementation" from the MMLU benchmark authors
– "HELM implementation" from Stanford
– "Harness implementation" from EleutherAI (recently updated – see the end of the thread) -
LLaMA Team Uses Original MMLU Benchmark Evaluation Code
By
–
3/ It turns out the LLaMA team used the original evaluation code proposed by the authors of the MMLU benchmark (find it at https://
github.com/hendrycks/test) Let's call it the "original implementation"