"FlashAttention-4" This new iteration of Flash Attention shows that on NVIDIA Blackwell GPUs the new bottleneck in Transformer attention isn’t matmul anymore but softmax + shared-memory traffic. So the latest designs stop treating attention as just GEMMs and start co-designing

→ View original post on X — @askalphaxiv