THE REVENGE OF PYTORCH
just kidding 🙂 @cHHillee (from PyTorch team) was kindly able to help improve the PyTorch baseline, done by 1) upgrading to nightly, 2) using the "compound" F.sdpa (scaled dot product attention) layer directly, and turning on a torch compile flag:

→ View original post on X — @karpathy,