Capacity and Trainability in Recurrent Neural Networks this very nice paper from 2016 (!) estimates certain types of RNNs can store 5 bits/param. they find this empirically and measure bits using the entropy of the distribution of class labels

→ View original post on X — @jxmnop