"Hyperball Optimization" Weight decay in normalized Transformers may be doing more than regularization, as it controls how quickly the model can change the direction of its weights. Hyperball does this directly by keeping weight norms fixed and normalizing every update, so
Hyperball Optimizes Transformers by Controlling Weight Direction Changes
By
–
