yeah… also harvey, hebbia, and all the benchmarks like MMLU, GPQA, HumanEval…
MACHINE LEARNING
-
AI Hallucinations Scale: Expertise Required for Detection
By
–
This is an important point – expertise & attention are required to figure out when an AI hallucinates, and the amount of effort required is increasing over time. But, models generally hallucinate less as they scale (with some exceptions), so net effect is complex, see medicine
-
Scaling in AI: Computing Power and Performance Growth
By
–
But the graph shows scaling works? I understand you want credit for your ideas here & as an academic I sympathize, but I am unqualified to adjudicate this dispute For most people (investors, safety folk, etc) scaling practically means “AI gets better with more computing power”
-
Scaling Reinforcement Learning: Intermediate Gains and Future Limitations
By
–
Scaling up RL is all the rage right now, I had a chat with a friend about it yesterday. I'm fairly certain RL will continue to yield more intermediate gains, but I also don't expect it to be the full story. RL is basically "hey this happened to go well (/poorly), let me slightly
-
Scaling Laws in LLMs: GPT-4 and Compute Optimization
By
–
Why do you keep doing this? I don’t understand the attempts to do childish gotcha moments. You are quoting me when GPT-4 was the best model & the labs were right. Scaling pre-training and inference compute both worked to make better models. Scaling has ALWAYS been logarithmic.
-
Compute Returns: 10x Compute Yields 10-30% Model Improvement
By
–
No one ever thought you would get 10x return for 10x compute. You get 10-30% returns for 10x compute. But if that improvement is enough to increase model ability in an economically meaningful way, it can be worth the cost.
-
Scaling Limitations and AI Labs’ Path to AGI
By
–
It feels dishonest to call it retconning when even the AI labs have been clear about diminishing returns. Some of them think that it won’t matter because they can still scale fast enough to AGI that the curve isn’t relevant. Or that other approaches like RL will boost scale.
-
Scaling Laws and Diminishing Returns in AI Development
By
–
I am not a good strawman here for you to gotcha quote tweet. I never predicted imminent AGI, but more importantly the entire point of the scaling law is diminishing returns to scale. Its a logarithmic curve, as has been known. That doesn’t mean you don’t get gains from scaling.
-
Sutton’s Bitter Lesson: Scaling Computation in Reinforcement Learning
By
–
The Bitter Lesson "Search and learning are general purpose methods that continue to scale with increased computation, even as the available computation becomes very great." — Richard Sutton Rich Sutton: http://
incompleteideas.net/IncIdeas/Bitte
rLesson.html
… #ReinforcementLearning