Most people don't understand that attention is a pooling operator. A much better (more FLOPs) and less hard-coded and peaky compared to "max" operator in the traditional max pooling, because of soft-attention with learned matmuls and dot products. However, attention and pooling

→ View original post on X — @aravsrinivas