The first step is to understand that when LMs are pre-trained on next-word prediction, they are really doing massive multi-task learning on thousands (millions?) of tasks. Here is a list of some potential tasks.
@_jasonwei
-
Predicting Next Word Reveals Arbitrary Tasks in Natural Text Distribution
By
–
Although the tasks above are well defined, the naturally occurring distribution of tasks in text turns out to be not well-defined. Here, I show that predicting the next word in a random sentence from Biden’s wikipedia page can include arbitrary tasks like comma prediction.
-
Whiteboard Lecture on Why Language Models Work Well
By
–
As a kid I loved whiteboard lectures way more than slides, so for Stanford’s CS25 class I gave a whiteboard lecture! My goal was to simply and clearly explain why language models work so well, purely via intuitions. Youtube video: https://
youtu.be/3gb-ZkVRemQ?si
=jvbzUmR9Q76PIc5r
… (w/ @hwchung27
) -
The Underappreciated Value of Evaluations Among Key Stakeholders
By
–
Great to see these evals! The interesting thing I have noticed is that evals are not valued by everyone as much as they should be. But the people who matter always value them intensely
-
Awesome Post on Devising ML Metrics by Dan
By
–
Just read, this is an awesome post Dan! (Originally thought it was just an image lol, here is the post: https://
safe.ai/blog/devising-
ml-metrics
…) -
Seven Sins of Language Model Evaluation: What Makes Evals Successful
By
–
New blog post where I discuss what makes an language model evaluation successful, and the "seven sins" that make hinder an eval from gaining traction in the community: https://
jasonwei.net/blog/evals Had fun presenting this at Stanford's NLP Seminar yesterday! -
Comprehensive Study on Predicting Language Model Performance
By
–
Enjoyed this extremely comprehensive study on predicting language model performance http://
arxiv.org/abs/2405.10938. Found many insightful nuggets:
– In a single model family there usually aren't that many model sizes, which hinders predictive power. However, there are many model -
Junior Engineers Can Contribute at Google Brain and Gemini
By
–
Yeah, this was not meant to be a dunk on google like some people are portraying it as. It is totally possible to do stuff as a junior engineer at Google, especially at Brain or Gemini. For instance, Quoc was super supportive of me and didn't care about my level. I'm just saying
-
GPQA High Variance Issues and Evaluation Methodology Discussion
By
–
I believe we're running 4 repeats and taking the average, and also using the same zero-shot CoT prompt as before, see https://
github.com/openai/simple-
evals
… But i totally agree with you that GPQA is high variance, I wish there were like 3k examples instead of 300 -
GPT-4o Achieves State-of-the-Art on GPQA Benchmark
By
–
Nice result lost in the hype: GPT-4o reclaims state-of-the-art on GPQA with 53.6%, overtaking Claude Opus (50.7%).