sure, the task would not be emergent *for that metric*, but its still emergent *for the metric we care about*
@_jasonwei
-
U-shaped Performance: Emergent Abilities in Model Scaling
By
–
Additional response 1: Many emergent abilities also cannot be explained by these arguments. Consider the below plots which show a U-shaped phenomena: performance actually decreases for several model scales, until it suddenly spikes up again.
-
Discussion on Different Metrics in Previous Research Paper
By
–
Finally, some discussion around using different metrics was given in our previous paper, which could be worth taking a look at: https://
openreview.net/pdf?id=yzkSU5z
dwD
… -
Chain-of-Thought Prompting: Emergence in Large Language Models
By
–
Additional response 2: Another popular example of emergence which also underscores qualitative changes in the model is chain-of-thought prompting, for which performance is worse than answering directly for small models, but much better than answering directly for large models.
-
Discussing Emergent Abilities in AI: Risks and Predictions
By
–
Overall, I’m glad that the idea of emergent abilities is being discussed more. I’m excited about work that would enable us to predict emergent behavior, since emergence includes risks as well as abilities. I’d love to discuss more with you on twitter or at the next conference!
-
Emergent Abilities Still Visible When Plotted
By
–
Response 2A: This is untrue, at least for many emergent abilities. You can plot it and still see the emergent spike.
-
Log-Scale X-Axis Best Represents Exponential Model Scaling
By
–
Response 2B: Log-scale x-axis is the best representation of how models improve, since we scale them exponentially.
-
Shayne Redford’s Flan Collection Accepted to ICML Conference
By
–
@ShayneRedford
's open-source "Flan Collection" work is accepted to ICML! As an intern at Google Brain, Shayne also co-first-authored Flan-T5 (which has 500k downloads/month on Huggingface btw), with a third paper coming out soon. This is quite prolific… -
Prioritizing Opportunity Cost Over Ideation in AI Development
By
–
AI now is special in that there is an abundance of ideas and potential projects. So coming up with new ideas is relatively less important than being good at deciding what is worth working on. "What's the opportunity cost of doing " should be part of every discussion.
-
Scaling Laws Enable Predictable Model Performance
By
–
I don't think so? "Scaling laws" states that perplexity is highly predictable. For example, GPT-4's loss on some evaluations can be predicted with models of less than 1,000x compute. (
https://
arxiv.org/pdf/2303.08774
.pdf
…)
Should be a clear difference between things that can be predicted with