25/ THE END PS: Please, come comment/discuss at https://
huggingface.co/spaces/Hugging
FaceH4/open_llm_leaderboard/discussions/82
… PPS: During the writing of this post, @Ollmer & the community have done an amazing work updating MMLU in the Harness to make it similar to the original. We're thus now updating the leaderboard based on it!
INNOVATION
-
MMLU Benchmark Updated in Open LLM Leaderboard
By
–
-
Open LLM Leaderboard: Evaluation Framework and Results
By
–
1/ First, what's the Open LLM Leaderboard? Just a wrapper running the @AiEleuther evaluation harness https://
github.com/EleutherAI/lm-
evaluation-harness
… on the Hugging Face cluster spare cycles + storing/displaying results at https://
huggingface.co/spaces/Hugging
FaceH4/open_llm_leaderboard
… -
LLaMA 65B MMLU Benchmark Discrepancy Analysis
By
–
2/ For one evaluation, MMLU (
https://
arxiv.org/abs/2009.03300), the community was surprised that the leaderboard numbers for the top model, LLaMA 65B, were significantly lower than the numbers in the published LLaMa paper: a 30% difference! We dived in a rabbit hole to understand -
ACL Initiative Broadens NLP Participation for Underrepresented Countries
By
–
Deadline approaching! This excellent initiative is meant to broaden participation in ACL in countries where our registration fees are way out of reach. Please spread the word & apply by June 28. #NLProc
-
Pooling Resources for Cooperative Large Model Training Infrastructure
By
–
Idea: who in the investment community wants to pool resources together to create a cooperative infrastructure for training large models? Feels wasteful to have startups burn VC money to train foundation models to do broadly the same thing.
-
Databricks acquires MosaicML in 1.3 billion generative AI deal
By
–
Monster outcome @NaveenGRao https://
wsj.com/articles/datab
ricks-strikes-1-3-billion-deal-for-generative-ai-startup-mosaicml-fdcefc06
… -
PyTorch Bridge: Minimal Code Changes for Lightning and HF Integration
By
–
Good point. It's essentially a stepping stone between PyTorch and the Lightning Trainer or HF Trainer. Lots of people have existing PyTorch code bases and don't want to do a bigger restructure of their code. So, here, you really only need to change a few lines.
-

AI Reveals Your Identity Through Writing Preferences
By
–
AI is incredible at reflecting back to you who you are. I keep a running list of writing I like and writers I admire in the Notes app on my phone. I threw them into ChatGPT and Claude to pull out patterns, and this is what came out:
-
Gemini: DeepMind Combines Language Models with Reinforcement Learning
By
–
The next @deepmind algorithm, Gemini, may combine a powerful language model with cutting-edge reinforcement learning, perhaps helping with planning and memory. That's one thing I learned from a very interesting chat with @demishassabis recently.
-
India showcases AI Clinical Decision Support System at G20
By
–
Highlighting India's progress in AI, Shekar Sivasubramanian, our CEO reflects on our partnership with @MoHFW_INDIA to build, deploy and showcase our transformative AI solutions like the Clinical Decision Support System to the world @g20org
. #G20India