When we derived SignSGD and Signum, it was surprising how just keeping the sign of stochastic gradients is sufficient for convergence and we could prove that theoretically. In practice, Signum and derivatives like Lion work well even at large scale. It is interesting to see Adam

→ View original post on X — @animaanandkumar