The deeper you go, the better it gets. At 6 layers: Grassmann trails by 15%
At 12 layers: Grassmann trails by only 11% The model learns to approximate richer interactions through depth and local multi-scale windows. This is proof of concept that attention isn't necessary.
Depth and local windows reduce Grassmann trails, replacing attention
By
–