Note that Llama and Mistral have parameterised norm layers, so weight decay has no regularizing effect
Weight Decay Ineffectiveness in Llama and Mistral Models
By
–
By
–
Note that Llama and Mistral have parameterised norm layers, so weight decay has no regularizing effect