(7/n) We test Sparse Iso-Parameter Scaling, where width and sparsity are increased, holding n_params constant. Using the standard practice (SP + dense optimal HPs), one would conclude only up to 50% sparsity help. Instead using SμPar allows an 87.5% sparse model to match dense

→ View original post on X — @cerebras