Some setup notes (cc @EthanJPerez
) – We used the exact 2-shot prompt for Quote Repetition, which is already U-shaped for Gopher/Chinchilla – We used fewer shots for Hindsight – We did few-shot instead of 0-shot for Negation QA – We also showed inverse scaling up to PaLM 62B
LLMS
-
Few-shot Prompt Setup Notes for Scaling Law Experiments
By
–
-
U-shaped Scaling and the Limitations of Inverse Scaling Tasks
By
–
Implications: 1. U-shaped scaling means that inverse scaling may not hold when extrapolated to larger models. 2. The term “inverse scaling task” is underspecified. A task can be inverse scaling for one type of prompting and positive or U-shaped for another type of prompting.
-
CoT Prompting Defends Against Inverse Scaling in Math Tasks
By
–
Second, we show that CoT prompting can defend against inverse scaling. For instance, CoT prompting achieves 100% on 7 out of 8 subtasks for Redefine Math.
-
U-shaped Scaling Behavior Emerges at Higher Computational Budgets
By
–
Our results first confirm inverse scaling behavior seen on prior models trained up to 500 zettaFLOPs. But at 2K zettaFLOPs, it becomes U-shaped. U-scaling has also been shown in prior work, such as BIG-Bench.
-
Inverse Scaling Becomes U-Shaped with Larger Language Models
By
–
New preprint!
— Jason Wei (@_jasonwei) 4 novembre 2022
By evaluating 5x larger language models, inverse scaling can become “U-shaped scaling”, which means that performance increases sharply after decreasing.
https://t.co/bZQndKqlB6
These two tasks here are Third Prize winners from the Inverse Scaling Prize. pic.twitter.com/8d3pu8DDrkNew preprint! By evaluating 5x larger language models, inverse scaling can become “U-shaped scaling”, which means that performance increases sharply after decreasing. https://
arxiv.org/abs/2211.02011 These two tasks here are Third Prize winners from the Inverse Scaling Prize. -
Meta AI Creates Largest Protein Language Model with 15B Parameters
By
–
Meta AI researchers trained a language model to fill in protein sequence gaps across millions of diverse proteins & scaled up to 15B parameters, creating the largest language model of proteins to date. More on our latest breakthrough in protein folding https://
bit.ly/3WoWcK2 -
Machine Learning GPT3 Website Free for Content Creators
By
–
Machine learning for deep thought: a GPT3-based website that every content creator should jump on while it’s free… https://
qburst.co -
ERNIE-Layout Achieves SOTA Results on Document Understanding Tasks
By
–
Introducing ERNIE-Layout, a layout knowledge enhanced document pre-training model that obtained SOTA results on 11 datasets and won 1st place on DocVQA (exceeding 90 for the 1st time) and WebSRC. Check Zero-shot Demo @huggingface
: http://
huggingface.co/spaces/PaddleP
addle/ERNIE-Layout
… -
Exploring Instruction Induction for Inferring AI Instruction Templates
By
–
Wonder how well it fares for longer instructions — I played around a while ago with using instruction induction to infer “instruction templates” from examples and it seemed to work:
-
Experimental Optimization of AI Instruction Prompts
By
–
Cool extension of Honovich et al.’s “instruction induction”, experimentally optimizing to find the best instruction prompt for a given set of task examples: https://t.co/h4Rd2A9JwY
— Riley Goodside (@goodside) 4 novembre 2022Cool extension of Honovich et al.’s “instruction induction”, experimentally optimizing to find the best instruction prompt for a given set of task examples: