Not saying that Transformers are worse than RNNs, mind you — Transformers are *the best* at *what deep learning does* (generalizing via interpolation), specifically *because* of their strongly interpolative architecture prior (MHA). They are, however, worse at learning symbolic

→ View original post on X — @fchollet,