If you want to know what an AI company truly cares about, just look at what evals they run. – Lots of academic evaluations against GPT-4 means pressure to train a model more capable than OpenAI.
– Companies that actually care about their consumer product will probably run lots
@_jasonwei
-
AI Companies’ True Priorities Revealed Through Their Evaluation Choices
By
–
-
Value of Informal Research Writing Over Polished Papers
By
–
I really enjoy reading informal write-ups and discussion threads from great researchers. Whereas published papers are a polished summary of final results, informal docs and discussions are raw—the researcher's thought process, working style, and negative experiments are revealed.
-
Guest lecture on intuitions for understanding large language models
By
–
It was an honor to give a guest lecture yesterday at Stanford’s CS330 class, "Deep Multi-Task and Meta-Learning"! I discussed a few very simple intuitions for how I personally think about large language models. Slides: https://
docs.google.com/presentation/d
/1hQUd3pF8_2Gr2Obc89LKjmHL0DlH-uof9M0yFVd3FA4/edit?usp=sharing
… Here are the six intuitions: (1) -
Proposal for Language Modeling Competition Between Humans and AI
By
–
Like the International Math Olympiad or Spelling Bee, there should be a “language modeling competition” where humans compete to predict the next word in a sequence. The best humans would probably still lose to GPT-2, and we’d have more empathy for how hard it is to be an LLM 🙂
-
Reinventing Yourself When Changing Companies: A New Optimization
By
–
One lesson that I learned from moving to OpenAI (which is applicable to changing companies generally) is that the opportunity to reinvent myself and adapt to a new optimization landscape can be fun. When I was at Google Brain from 2020-2022, my optimization was simply writing
-
ChatGPT Recognition Becomes New Status Symbol Metric
By
–
"Known by ChatGPT" will be the new "has a wikipedia page"
-
Aspirations for Greatness in AI Research Like Sports
By
–
I tried to give this talk in the spirit of "a college soccer player watches videos of Messi and analyzes what makes him such a great soccer player." IMO it's great for people to aspire for greatness in AI research, just like in sports. Sorry you took it personally.
-
AI Research Talk at UC Berkeley: Key Trends in Researcher Success
By
–
Enjoyed visiting UC Berkeley’s Machine Learning Club yesterday, where I gave a talk on doing AI research. Slides: https://
docs.google.com/presentation/d
/1p1N1an6wjmjvXHCwiOwJJq6D58iINrceI9-ZkpmJwrY/edit?usp=sharing
… In the past few years I’ve worked with and observed some extremely talented researchers, and these are the trends I’ve noticed: 1. When -
Four criteria for evaluating new prompting techniques adoption
By
–
For any new prompting technique (e.g., tree-of-thought, least-to-most, graph-of-thoughts prompting), I consider four things to decide if it will become widely adopted: 1. How easy is it to implement
2. How much compute it use
3. How many tasks does it improve
4. How much does it