Although the bitter lesson suggests that end-to-end eventually wins, this talk observes that at a given level of (compute, data, algorithm, architecture), there exists an optimal structure to add to make things work at all. But we often forget to remove these when more end-to-end
@_jasonwei
-
Lecture appreciation and recommendation for Hyung Won’s segment
By
–
Hope the lecture is helpful, and thanks @stevenyfeng @SeongHee17633 @DivGarg9 @EmilyBunna for having me! Also check out Hyung Won's second half of the lecture, which is about inductive biases and how AI has become more end to end
-
Plot Scaling Laws to Predict Long-Term Performance Improvement
By
–
I end with general advice to plot scaling laws for your work. Scaling laws will tell you whether doing more of your thing (on some axis) will continue improving performance or whether it will saturate (i.e., whether it will stand the test of time).
-
Task Decomposition and Emerging Model Capabilities Through Subtasks
By
–
Often these tasks are a decomposition of multiple sub-tasks, and changes in behavior can be explained by the ability for models to do new subtasks.
-
Inverse Scaling and U-Shaped Performance in Language Models
By
–
One interesting phenomena is that performance for some tasks can get worse with scale (“inverse scaling”), then potentially get better ("U-shaped scaling"). This is a nice case study to help understand language model behavior.
-
Model Size and Emergence of Specialized Capabilities in AI
By
–
For harder tasks OTOH, only a large enough model will be incentivized to improve their loss (because by definition, they are harder). For example, GPT-4 is way better at math or physics than GPT 3.5. These abilities might appear only when you reach a certain model size.
-
Task Performance Decomposition: Smooth and Sudden Scaling Patterns
By
–
And it turns out that you can take a corpus of tasks, for which the average performance scales smoothly, and decompose them into tasks that increase smoothly or suddenly. There are many tasks in both categories.
-
Multi-task Learning in Next-word Prediction: Task-specific Loss Dynamics
By
–
Now given that next-word prediction is multi-task learning, we can write the overall loss is the weighted sum of loss of individual tasks. When overall loss improves smoothly, do all individual tasks improve smoothly, or do some improve at different rates than others?
-
Different Tasks Improve at Different Rates with AI Models
By
–
It seems obvious that different tasks should improve at different rates. For easy tasks, it is likely that they don’t improve much after we saturate them. For example, GPT-3.5 is basically as good as GPT-4 at writing grammatical sentences or classifying sentiment analysis.
-
Scaling Laws: How Compute Investment Improves Model Performance
By
–
Next, “scaling laws” assert that overall loss is expected to improve as you scale up the compute used to train your model. This was the motivation for our current scaling paradigm—as you invest more in scale, your model reliably gets better instead of plateauing.