Yeah, instruction tuning is kinda like "better data"
@_jasonwei
-
Finetuning vs Emergence: Different Paths to Model Abilities
By
–
Finetuning is a great way to bring an ability to a smaller model. Emergence often occurs for an ability that is *not* explicitly trained for, just by increasing scale (compute/ model size).
-
Survey Recommendation for Field Newcomers
By
–
a nicely put together survey for people entering the field!
-
Jason Wei Joins OpenAI ChatGPT Team from Google Brain
By
–
Personal update: after two years at @Google Brain, I joined the #ChatGPT team at @OpenAI
! Excited to continue working on large language models and can't wait to see the impact of AI on society -
Predicting Large Model Performance Across Unseen Tasks
By
–
(3 cont.) Can you predict the performance of one model from another?
Can you predict the performance of a 128B model on an unseen task, given models up to some smaller threshold size and some performances of 128B models on other tasks? -
Hope for Better Open-Source Large Language Models
By
–
Finally, with models like OPT and BLOOM at 175B params, I'm hopeful that better open-source models will be soon be trained, given the pace of innovation in the space.
-
BIG-Bench metrics deserve deeper analysis and study
By
–
(3) Even just looking at BIG-Bench metrics is quite understudied IMO. There are hundreds of tasks in BIG-Bench, and each task has dozens of models evaluated, each with many evaluation metrics. There are task logs for some models. This raises natural questions:
-
Flan-T5 Public Release with Instruction Tuning Data Ablation
By
–
(2 cont.) Flan-t5 is publicly available along with the dataset (thanks to @ShayneRedford
), so you can even ablate parts of the instruction tuning data to see what unlocks performance. -
Unlocking emergent abilities in smaller language models like Flan-T5
By
–
(2) One key question of emergence is how to unlock such abilities at smaller scales, and I don't think there has been much work on flan-T5, which often beats PaLM 62B as shown in this paper: https://
arxiv.org/abs/2210.11416 -
Scaling Unlocks Emergent Abilities in Language Models Talk
By
–
I gave a version of this talk "Scaling unlocks emergent abilities in language models" today at USC ISI. There is a video recording: https://
youtu.be/Z_Qt737HG-0 Thanks Justin Cho @HJCH0 for inviting me and organizing!