By
–
ChatGPT Plus users can use Browsing Mode to navigate around paywalled articles: Tested this on a paywalled article from Fortune Magazine. After GPT-4 Browsing printed the article, I paid for the Fortune subscription to confirm – it was the same text. ChatGPT did not
By
–
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond This survey paper provides a comprehensive and practical guide on working with LLMs for various settings(in-context learning & fine-tuning). I like how it approach LLMs usage in lens of models, data, and
By
–
Free Generative AI Conference on LLMs and AI Agents! July 11. We'll discuss building custom LLMs, simple AI agents, and the future of generative AI. You can register now: https://
eventbrite.com/e/stateofthear
t-free-generative-ai-conference-on-llms-and-ai-agents-tickets-641826449187
….
By
–
This was a really cool project with @clefourrier @nathanhabib1011 @osanseviero and a lot of help from @slippylolo from the Falcon team and the amazing LLaMA team, in particular Xavier Martinet, Aurélien Rodriguez and Sharan Narang!
By
–
25/ THE END PS: Please, come comment/discuss at https://
huggingface.co/spaces/Hugging
FaceH4/open_llm_leaderboard/discussions/82
… PPS: During the writing of this post, @Ollmer & the community have done an amazing work updating MMLU in the Harness to make it similar to the original. We're thus now updating the leaderboard based on it!
By
–
24/ That's why open, standardized, reproducible benchmarks such as the EleutherAI Harness https://
github.com/EleutherAI/lm-
evaluation-harness/
… or Stanford HELM https://
github.com/stanford-crfm/
helm/
… are invaluable to the community. Without them comparing results across models/papers would be impossible, stifling research!
By
–
23/ …not at all comparable even if they're both called MMLU & evaluated on same dataset Takeaway? Evaluations are strongly tied to implementations–down to minute details. A mere indication of "MMLU score" gives almost no information about how you can compare these numbers
By
–
22/ Say you've trained a perfect LLaMA 65B reproduction & evaluated it with EAI harness (score 0.488). Comparing it to the published number (evaluated w. original implementation, score 0.637), it's a 30% difference so you're likely thinking "Oh no " But these numbers are…
By
–
19/ In this case, we use probabilities again but this time we compute the probability of generating the full answer sequence, not just the letter: we sum the log of the probabilities and compute a normalization by dividing by the number of tokens to not penalize longer sequences.
By
–
18/ Harness: Now we finally turn to the EleutherAI Harness implementation (as of January 2023) which was used to compute the numbers for the Open LLM Leaderboard. Here is yet another way to compute a score for the model on the very same evaluation dataset! Let's take a look: