Weight decay is usually presented as “encouraging simpler solutions”, but I tend to think that the real benefit is the soft pruning of noisy / unhelpful features. Without decay, a weight can random-walk to a large value even if the input is completely random. Momentum and

→ View original post on X — @id_aa_carmack,