1/ First, what's the Open LLM Leaderboard? Just a wrapper running the @AiEleuther evaluation harness https://
github.com/EleutherAI/lm-
evaluation-harness
… on the Hugging Face cluster spare cycles + storing/displaying results at https://
huggingface.co/spaces/Hugging
FaceH4/open_llm_leaderboard
…
AI
-
Open LLM Leaderboard: Evaluation Framework and Results
By
–
-
LLaMA 65B MMLU Benchmark Discrepancy Analysis
By
–
2/ For one evaluation, MMLU (
https://
arxiv.org/abs/2009.03300), the community was surprised that the leaderboard numbers for the top model, LLaMA 65B, were significantly lower than the numbers in the published LLaMa paper: a 30% difference! We dived in a rabbit hole to understand -
Open LLM Leaderboard evaluation accuracy analysis findings
By
–
A on "what was going on with the Open LLM Leaderboard?" its numbers didn't match the ones reported in LLaMA paper so we dived in it and wrote a blog post of learnings! Here's the thread version for those of you who didn't want to read a blog post
-
ACL Initiative Broadens NLP Participation for Underrepresented Countries
By
–
Deadline approaching! This excellent initiative is meant to broaden participation in ACL in countries where our registration fees are way out of reach. Please spread the word & apply by June 28. #NLProc
-
Pooling Resources for Cooperative Large Model Training Infrastructure
By
–
Idea: who in the investment community wants to pool resources together to create a cooperative infrastructure for training large models? Feels wasteful to have startups burn VC money to train foundation models to do broadly the same thing.
-
AI Regulation: Benefits for Business and Policymakers
By
–
With the arrival of #GenerativeAI, it’s increasingly critical for governments to promote the responsible use of AI. Global AI Ethicist, @HaniyehMah, shares her insights with @CNBCnow on how AI regulations can benefit both businesses and policymakers. https://
cnb.cx/43076rG -
Databricks acquires MosaicML in 1.3 billion generative AI deal
By
–
Monster outcome @NaveenGRao https://
wsj.com/articles/datab
ricks-strikes-1-3-billion-deal-for-generative-ai-startup-mosaicml-fdcefc06
… -
PyTorch Bridge: Minimal Code Changes for Lightning and HF Integration
By
–
Good point. It's essentially a stepping stone between PyTorch and the Lightning Trainer or HF Trainer. Lots of people have existing PyTorch code bases and don't want to do a bigger restructure of their code. So, here, you really only need to change a few lines.
-
LLMs Demonstrate Powerful Investigation and Analysis Capabilities
By
–
There is something so powerful about large language models’ abilities here. I could have commissioned an entire team of investigators to attempt to piece something like this together about me, given them months of time and thousands of dollars, and not gotten anything close to
-
AI Reveals Your Identity Through Writing Preferences
By
–
AI is incredible at reflecting back to you who you are. I keep a running list of writing I like and writers I admire in the Notes app on my phone. I threw them into ChatGPT and Claude to pull out patterns, and this is what came out: