22/ Say you've trained a perfect LLaMA 65B reproduction & evaluated it with EAI harness (score 0.488). Comparing it to the published number (evaluated w. original implementation, score 0.637), it's a 30% difference so you're likely thinking "Oh no " But these numbers are…
@thom_wolf
-
Model Evaluation Methods Significantly Impact Scores and Rankings
By
–
21/ Now let’s compare the model scores on these three possible ways to evaluate the models We can see that for the same dataset, both absolute scores and model rankings are very sensitive to the evaluation method we decide to use.
-
Probability-based sequence generation normalization in language models
By
–
19/ In this case, we use probabilities again but this time we compute the probability of generating the full answer sequence, not just the letter: we sum the log of the probabilities and compute a normalization by dividing by the number of tokens to not penalize longer sequences.
-
Harness Evaluation Framework for Large Language Models
By
–
18/ Harness: Now we finally turn to the EleutherAI Harness implementation (as of January 2023) which was used to compute the numbers for the Open LLM Leaderboard. Here is yet another way to compute a score for the model on the very same evaluation dataset! Let's take a look:
-
HELM Implementation: Model Evaluation via Token Probabilities
By
–
16/ HELM: Now let's take a look at the HELM implementation. The few-shot prompt is similar but the way the model is evaluated is quite different: we use the next token probabilities from the model to select a text generation and we compare it to the text of the expected answer
-
Few-Shot Prompting Improves Model Performance on MMLU
By
–
15/ To make sure that the model does as few as possible of these sort of errors, we can use “few shots”: we provide the model with a few examples in the prompt with the expected answers. Since this improves performance, MMLU is evaluated in 5 shots in all our experiments
-
Model Evaluation: Uncounted Prediction Errors in Multiple Choice
By
–
14/ As you can see, we compare the probabilities predicted by the model, on the four answers *only*. But sometimes that model would have made a mistake (generating "Zygote" here) which is not counted as a mistake…
-
Two main ways to evaluate model outputs
By
–
11/ In summary, we have two main ways to evaluate a model outputs:
1. Get the probabilities that some tokens are continuations of the prompt => compare these probabilities together
2. Get the text generation from the model => compare these text generations to possible choices -
Token Selection and Autoregressive Generation in Language Models
By
–
10/ We can use these probabilities to choose a token, for instance the most probable. Adding the selected token to the prompt + feeding it back to the model allows to generate another token and so on until whole sentences are created as continuations of the input prompt.
-
Three Major MMLU Evaluation Codebases Compared
By
–
5/ We thus have (at least) 3 serious codebases for evaluating on the same MMLU dataset:
– "Original implementation" from the MMLU benchmark authors
– "HELM implementation" from Stanford
– "Harness implementation" from EleutherAI (recently updated – see the end of the thread)