Span corruption in t5 is just a data operation over regular text. Masking stuff in inputs and moving them to targets. It's still fundamentally autoregressive loss on the decoder end.

→ View original post on X — @yitayml,