There are two ways to get to superhuman AI: improving AI or declining human intelligence
@yitayml
-
T5X and SeqIO remain dominant LLM training frameworks
By
–
Looks good! Would love to play with this someday. Although I think it would be hard to dethrone t5x + seqio as the best LLM training framework.
-
Behind-the-scenes training details of Flan-T5 model revealed
By
–
Insightful thread by @hwchung27 that shares some behind the scenes of training flan-t5. These details are often overlooked and sometimes mentioned in footnotes in papers (or not mentioned at all, i.e., "not paper worthy"). Sometimes they are stack dependent, model dependent or
-
FastChat-T5 Encoder-Decoder Model Performs Well on Chatbot Arena
By
–
Yeah but fastchat-t5 ranking decently for it's size on chatbot arena is a pretty good evidence of encoder decoder models being able to actually chat well.
-
Training T5 Model on Language Modeling Objective
By
–
Training t5 on lm objective for 100k more steps. There are lm adapted ckpts of t5 on the internet iirc.
-
Tokenizer Effects on Math Tasks: T5 vs LM Adaptation
By
–
Math tasks could be due to c4 sentencepiece/tokenizer though. T5 can't open end generate that well due to span corruption objective but LM adaptation or ul2 fixes this
-
Open Source Model Evaluation Results Exceed Public Competition
By
–
thanks for running the evals. indeed the numbers look great (and promising). yeah i think it's scalable and efficient. like your tweet suggested it's a pity there's no t5 or ul2 model >40B. otherwise i think it would beat every public open source model right now.
-
PaLI-X Combines ViT-22B Vision and Multilingual UL2-32B
By
–
Check out PaLI-X that combines ViT-22B and multilingual UL2-32B!
-
Science Over Hype: Debunking AI Claims
By
–
this is not about promotion. it's about science and hype busting.
-
Encoder-Decoder vs Decoder-Only Models: T5 Paper Guide
By
–
Additional reading for people confused about encoder-decoder models vs decoder-only models: most wisdom is also from the 2019 T5 paper. it's the bible for NLP literally.