oh yeah! That was Pretraining without attention: https://
arxiv.org/abs/2212.10544

→ View original post on X — @jxmnop