24/ That's why open, standardized, reproducible benchmarks such as the EleutherAI Harness https://
github.com/EleutherAI/lm-
evaluation-harness/
… or Stanford HELM https://
github.com/stanford-crfm/
helm/
… are invaluable to the community. Without them comparing results across models/papers would be impossible, stifling research!
OPEN SOURCE
-
Open Standardized Benchmarks Essential for AI Model Evaluation
By
–
-
LLaMA 65B Evaluation Discrepancies Between Implementations
By
–
22/ Say you've trained a perfect LLaMA 65B reproduction & evaluated it with EAI harness (score 0.488). Comparing it to the published number (evaluated w. original implementation, score 0.637), it's a 30% difference so you're likely thinking "Oh no " But these numbers are…
-
Harness Evaluation Framework for Large Language Models
By
–
18/ Harness: Now we finally turn to the EleutherAI Harness implementation (as of January 2023) which was used to compute the numbers for the Open LLM Leaderboard. Here is yet another way to compute a score for the model on the very same evaluation dataset! Let's take a look:
-
HELM Benchmark Implementation Analysis for MMLU Evaluation
By
–
4/ Diving further, we found yet another serious implementation for evaluating on the very same MMLU dataset: the code used in the HELM benchmark https://
crfm.stanford.edu from @StanfordCRFM
: https://
github.com/stanford-crfm/
helm
… Let's call it the "HELM implementation" -
Open LLM Leaderboard: Evaluation Framework and Results
By
–
1/ First, what's the Open LLM Leaderboard? Just a wrapper running the @AiEleuther evaluation harness https://
github.com/EleutherAI/lm-
evaluation-harness
… on the Hugging Face cluster spare cycles + storing/displaying results at https://
huggingface.co/spaces/Hugging
FaceH4/open_llm_leaderboard
… -
LLaMA 65B MMLU Benchmark Discrepancy Analysis
By
–
2/ For one evaluation, MMLU (
https://
arxiv.org/abs/2009.03300), the community was surprised that the leaderboard numbers for the top model, LLaMA 65B, were significantly lower than the numbers in the published LLaMa paper: a 30% difference! We dived in a rabbit hole to understand -
Open LLM Leaderboard evaluation accuracy analysis findings
By
–
A on "what was going on with the Open LLM Leaderboard?" its numbers didn't match the ones reported in LLaMA paper so we dived in it and wrote a blog post of learnings! Here's the thread version for those of you who didn't want to read a blog post
-
PyTorch Bridge: Minimal Code Changes for Lightning and HF Integration
By
–
Good point. It's essentially a stepping stone between PyTorch and the Lightning Trainer or HF Trainer. Lots of people have existing PyTorch code bases and don't want to do a bigger restructure of their code. So, here, you really only need to change a few lines.
-
DragGAN Source Code and Demo Now Available Online
By
–
DragGAN source code and demo available now! 💥
— abhishek (@abhi1thakur) 26 juin 2023
Try it here: https://t.co/WJgFgc617J
Code: https://t.co/BFWpUS9HaB pic.twitter.com/Xiw6R80dWpDragGAN source code and demo available now! Try it here: https://
huggingface.co/spaces/radames
/DragGan
…
Code: https://
github.com/XingangPan/Dra
gGAN
… -
Top AI Models and Datasets Weekly Trends Highlighted
By
–
Trending models, datasets, spaces and PAPERS (
https://
huggingface.co/Papers!!) of the week on http://
hf.co. Congrats to @TIIuae (falcon) @MosaicML (mpt) @StabilityAI (SDXl) @MetaAI (MusicGen) @MSFTResearch (Textbooks AAYN) @UMBaltimore (through your eyes) and many others!