Thank you for tweeting our paper. This is the actual TLDR: We design the first deterministic initialization that works with any network architecture. This includes identity initialization when weight matrix is square and Hadamard initialization when weight matrix is rectangular.

→ View original post on X — @animaanandkumar,