Check out OPT-IML, but a lot of people said they didn't know:
– Flan-T5 checkpoints are publicly available (without requesting access)
– Flan-T5 11B outperforms OPT-IML on MMLU and Big-Bench Hard, despite being 10x more compute efficient Checkpoints: https://
huggingface.co/docs/transform
ers/model_doc/flan-t5
…
@_jasonwei
-
Flan-T5 Outperforms OPT-IML with Greater Efficiency
By
–
-
Flan-T5 Outperforms T5 in Finetuning Efficiency
By
–
Key takeaway: finetuning Flan-T5 is better and more compute-efficient than finetuning T5. In other words, Flan-T5 > T5 for every real scenario I can think of. Don't use the pre-trained checkpoint—always finetune!
-
U-shaped Scaling: Model Performance Recovery in Larger Language Models
By
–
Another newer example of emergence that I’m excited about is U-shaped scaling, where even if model performance goes down from small → medium models, it can go back up for large models. See our paper: https://
x.com/_jasonwei/stat
us/1588605909781319680
… Inverse scaling benchmark: -
Emergence as Framework for Understanding Language Model Scaling
By
–
Overall, emergence is nice framework for viewing language models
– Emergent abilities cannot be predicted via scaling plots for small models
– Keep scaling, and we might see more emergent abilities 🙂 See our TMLR 2022 piece (w/ survey certification): -
Instruction Tuning as an Emergent Ability in Large Language Models
By
–
One of the most interesting emergent abilities IMO is instruction tuning. Anthropic and Flan-LaMDA suggest that zero-shot performance can improve from RLHF and NLP benchmark instruction tuning (although text-davinci usually loses to code-davinci). https://
arxiv.org/abs/2204.05862 -
Emergent Abilities in Large Language Models Over Past Year
By
–
Throughout the past year, there have been hundreds of emergent abilities, which can only be observed in large-enough language models. I previously made a list of them (more than 100):https://t.co/QLfwNUeIBx
— Jason Wei (@_jasonwei) 25 janvier 2023Throughout the past year, there have been hundreds of emergent abilities, which can only be observed in large-enough language models. I previously made a list of them (more than 100):
-
Emergent Abilities in Language Models Through Scaling
By
–
Yesterday I gave a lecture at @Stanford
's CS25 class on Transformers! The lecture was on how “emergent abilities” are unlocked by scaling up language models. Emergence is one of the most exciting phenomena in large LMs… Slides: -
Excitement for 2023 Language AI Research and Team Achievements
By
–
Delighted to see the work from our team (chain-of-thought prompting, emergent abilities, self-consistency, med-palm) featured here, and super excited for language AI research to continue in 2023!
-
Comparing Text Davinci 2 and 3 on NLP Benchmarks
By
–
We compared text davinci 2 and text davinci 3 on a bunch of nlp benchmarks, see here:
-
Emergence in Language Models: Scaling and Ability Discovery
By
–
Key questions of emergence in language models:
– Why does scaling unlock emergent abilities?
– What is the full range of emergent abilities in language models, and how can we discover them systematically?