This is the training degeneracy that @jiawzhao is talking about if you only do identity initialization and not Hadamard for rectangular weight matrices
@animaanandkumar
-
Hadamard Transform Essential for Dimension Changes and Degeneracy
By
–
The dimension changes need Hadamard transform and without it you have bad degeneracy. We discuss this in detail
-
Deterministic Initialization for Any Network Architecture
By
–
Thank you for tweeting our paper. This is the actual TLDR: We design the first deterministic initialization that works with any network architecture. This includes identity initialization when weight matrix is square and Hadamard initialization when weight matrix is rectangular.
-
Surgical AI Documentary: Gesture Detection and Surgeon Training
By
–
Great documentary on Surgical AI work we are doing together @AjhungMD Come work with us on transformative Surgical AI that can detect surgical gestures, warn surgeons before a mistake is made and ultimately help them train to be better. https://
cms.caltech.edu/about/position
s/surgicalai
… -
Neural Operators and AI for Science Talk at MIT IAIFI
By
–
It was great visiting @mit @iaifi_news and give a talk about neural operators and AI for science. You can find my talk here https://
youtu.be/RR5-mYQOb7E?t=
1208
…