Math tasks could be due to c4 sentencepiece/tokenizer though. T5 can't open end generate that well due to span corruption objective but LM adaptation or ul2 fixes this

→ View original post on X — @yitayml