Another by Yann LeCun! “The Spike, the Sparse and the Sink” This paper shows that massive activations and attention sinks come from the same pre-norm Transformer pipeline rather than being separate anomalies. Early SwiGLU blocks act like directional quadratic amplifiers,

→ View original post on X — @askalphaxiv