Best is to evaluate the models with a prompt where the understand the task. If a model is 20% below with a prompt compared to another, clearly you are not evaluating the model properly and it makes no sense to report this number or use it in a comparaison.
@guillaumelample
-
Prompt engineering investigation: minimal sensitivity to formatting changes
By
–
We spent a lot of time investigating what others papers did (Chinchilla, GPT-3, PaLM) but very few of them actually provide any prompt so we just implemented what made sense to us. And we did not observe 20% differences by adding or removing a space in the prompt.
-
Custom Prompts and Benchmark Evaluation Standards for LLMs
By
–
You mention our "custom prompt" like if there was an official way of prompting (yours?). Most benchmarks were created before this concept of LLM eval with prompting even exists, and for many of them there is no official prompt or way to evaluate them with LLMs.
-
Formatting Alone Cannot Explain Performance Gap with LLaMA
By
–
I don't think that formatting alone can explain such a gap. You could maybe try to reproduce LLaMA numbers given the available model. It is likely that fixing these differences for LLaMA will also improve the numbers you report for your model.
-
LLaMA evaluation metrics concern and measurement discrepancies
By
–
It is true that accuracy on some metrics can be quite sensitive to the prompt, however this is not normal that all metrics reported for LLaMA here are systematically (and significantly) below what we measured. There may be an issue in how LLaMA was evaluated.
-
Prompt Technique pour Interaction avec LLaMA sans Fine-tuning
By
–
Clever prompt that enables chat interaction with the raw version of LLaMA, without requiring any fine-tuning of the model on instruction data !
-
Rationale for Creating Carper as Stability Eleuther Subset
By
–
I see. I understand for Stability/Eleuther, but what was the rationale behind creating Carper if this is merely a subset?
-
Differences between organizations training LLMs and their collaboration
By
–
And in practice what is the difference? Do they collaborate / do they share compute? I have seen people from each of these organisations saying they are training some LLMs, are they referring to the same models?
-
Differences between Stability.ai, Eleuther.ai, and Carper.ai
By
–
What is the difference between http://
Stability.ai / http://
Eleuther.ai / http://
Carper.ai ? I always get confused.