SGD is a ResNet too (the blocks of it are fwd+bwd), the residual stream is the weights so… We're not taking the Attention is All You Need part literally enough? 😀
SGD as ResNet: Rethinking Attention Architecture Fundamentals
By
–
By
–
SGD is a ResNet too (the blocks of it are fwd+bwd), the residual stream is the weights so… We're not taking the Attention is All You Need part literally enough? 😀