Unlocking reasoning in smaller language models is a great direction
@_jasonwei
-
Important Research Beyond Simple Model Scaling
By
–
This blog post explains an important research that does not involve more scaling
-
Smaller Models with Better Data Can Outperform Larger Ones
By
–
Great point. We are seeing more and more than smaller models with better objectives or data can beat big ones! My main point is that an approach shouldnt go away as models get better. Scale is just one way of getting better
-
Progress in Model Training: From Full Dataset Collection to Modern Approaches
By
–
Definitely a limitation. I see some
progress, though. Back in the day, we had to collect an entire training dataset and train a new model! -
Recommendation to Follow Le Hou’s Research on Large Language Models
By
–
People interested in large LMs should follow Le Hou (
@Hou_Le
) at @GoogleAI
, who made a new twitter account recently Le has done great work such as Flan, and self-play for reasoning (
https://
arxiv.org/abs/2210.11610). I'm sure we'll see more great work from him 🙂 -
Few-shot Prompt Setup Notes for Scaling Law Experiments
By
–
Some setup notes (cc @EthanJPerez
) – We used the exact 2-shot prompt for Quote Repetition, which is already U-shaped for Gopher/Chinchilla – We used fewer shots for Hindsight – We did few-shot instead of 0-shot for Negation QA – We also showed inverse scaling up to PaLM 62B -
U-shaped Scaling and the Limitations of Inverse Scaling Tasks
By
–
Implications: 1. U-shaped scaling means that inverse scaling may not hold when extrapolated to larger models. 2. The term “inverse scaling task” is underspecified. A task can be inverse scaling for one type of prompting and positive or U-shaped for another type of prompting.
-
Inverse Scaling Prize Round 2 Evaluation Announcement
By
–
Make sure to check out the inverse scaling prize, which is a great community effort! Looking forward to evaluating on the Round 2 winners 🙂
-
CoT Prompting Defends Against Inverse Scaling in Math Tasks
By
–
Second, we show that CoT prompting can defend against inverse scaling. For instance, CoT prompting achieves 100% on 7 out of 8 subtasks for Redefine Math.