Love the idea, fascinating design! But wait, it's almost 3.5x the number of extra active parameters? Doesn't that violate rule 2?
Model Design Trade-offs: Parameter Count and Efficiency Rules
By
–
By
–
Love the idea, fascinating design! But wait, it's almost 3.5x the number of extra active parameters? Doesn't that violate rule 2?