We developed the first fully deterministic weight initialization procedure for training any neural network. Our scheme, ZerO, initializes networks’ weights with only zeros and ones based on identity, and Hadamard transforms. ZerO achieves SOTA https://
arxiv.org/abs/2110.12661

→ View original post on X — @animaanandkumar