16/ HELM: Now let's take a look at the HELM implementation. The few-shot prompt is similar but the way the model is evaluated is quite different: we use the next token probabilities from the model to select a text generation and we compare it to the text of the expected answer
LLMS
-
Few-Shot Prompting Improves Model Performance on MMLU
By
–
15/ To make sure that the model does as few as possible of these sort of errors, we can use “few shots”: we provide the model with a few examples in the prompt with the expected answers. Since this improves performance, MMLU is evaluated in 5 shots in all our experiments
-
Two main ways to evaluate model outputs
By
–
11/ In summary, we have two main ways to evaluate a model outputs:
1. Get the probabilities that some tokens are continuations of the prompt => compare these probabilities together
2. Get the text generation from the model => compare these text generations to possible choices -
Token Selection and Autoregressive Generation in Language Models
By
–
10/ We can use these probabilities to choose a token, for instance the most probable. Adding the selected token to the prompt + feeding it back to the model allows to generate another token and so on until whole sentences are created as continuations of the input prompt.
-
Three Major MMLU Evaluation Codebases Compared
By
–
5/ We thus have (at least) 3 serious codebases for evaluating on the same MMLU dataset:
– "Original implementation" from the MMLU benchmark authors
– "HELM implementation" from Stanford
– "Harness implementation" from EleutherAI (recently updated – see the end of the thread) -
LLaMA Team Uses Original MMLU Benchmark Evaluation Code
By
–
3/ It turns out the LLaMA team used the original evaluation code proposed by the authors of the MMLU benchmark (find it at https://
github.com/hendrycks/test) Let's call it the "original implementation" -
LLaMA 65B MMLU Benchmark Discrepancy Analysis
By
–
2/ For one evaluation, MMLU (
https://
arxiv.org/abs/2009.03300), the community was surprised that the leaderboard numbers for the top model, LLaMA 65B, were significantly lower than the numbers in the published LLaMa paper: a 30% difference! We dived in a rabbit hole to understand -
Open LLM Leaderboard: Evaluation Framework and Results
By
–
1/ First, what's the Open LLM Leaderboard? Just a wrapper running the @AiEleuther evaluation harness https://
github.com/EleutherAI/lm-
evaluation-harness
… on the Hugging Face cluster spare cycles + storing/displaying results at https://
huggingface.co/spaces/Hugging
FaceH4/open_llm_leaderboard
… -
Open LLM Leaderboard evaluation accuracy analysis findings
By
–
A on "what was going on with the Open LLM Leaderboard?" its numbers didn't match the ones reported in LLaMA paper so we dived in it and wrote a blog post of learnings! Here's the thread version for those of you who didn't want to read a blog post
-
Pooling Resources for Cooperative Large Model Training Infrastructure
By
–
Idea: who in the investment community wants to pool resources together to create a cooperative infrastructure for training large models? Feels wasteful to have startups burn VC money to train foundation models to do broadly the same thing.