Our results first confirm inverse scaling behavior seen on prior models trained up to 500 zettaFLOPs. But at 2K zettaFLOPs, it becomes U-shaped. U-scaling has also been shown in prior work, such as BIG-Bench.
@_jasonwei
-
Inverse Scaling Becomes U-Shaped with Larger Language Models
By
–
New preprint!
— Jason Wei (@_jasonwei) 4 novembre 2022
By evaluating 5x larger language models, inverse scaling can become “U-shaped scaling”, which means that performance increases sharply after decreasing.
https://t.co/bZQndKqlB6
These two tasks here are Third Prize winners from the Inverse Scaling Prize. pic.twitter.com/8d3pu8DDrkNew preprint! By evaluating 5x larger language models, inverse scaling can become “U-shaped scaling”, which means that performance increases sharply after decreasing. https://
arxiv.org/abs/2211.02011 These two tasks here are Third Prize winners from the Inverse Scaling Prize. -
Inverse Scaling Observed Up to 62B Model Parameters
By
–
We also showed inverse scaling up to 62B on our model / prompt setup