The world is made of mutual information. Entropy is just a solipsistic instance of it (the mutual information between a variable and itself).
@pmddomingos
-
Causality as a subtree in the taxonomy of inductive biases
By
–
Agreed. From an ML point of view, there's a taxonomy of inductive biases, and causality is a subtree.
-
Designer’s Choice: Local vs Global Translation Invariance in ConvNets
By
–
It's the designer's choice (that's why I said "or"), and that's a feature, not a bug. But you also need to distinguish between local and global invariance. All convnets have some degree of translation invariance, or they wouldn't be convnets.
-
Remembering Finite State Probabilities in the Real World
By
–
You just remember the probability of each state (of which in the real world there is always only a finite number).
-
Inductive Biases and Prior Assumptions in Machine Learning
By
–
With inductive biases, i.e., prior assumptions that may or may not be causal.
-
ConvNets Translation-Invariance: Built-in Ability and Learning Limitations
By
–
To be precise, convnets have the built-in ability to be translation-invariant, ignoring image-boundary effects. They can also learn partly or non-invariant features, which is good, but depending on the data may result in not learning invariances that are present.
-
Training data presence affects image generation result interest
By
–
It is possible that images like this were in the training set, in which case the result is indeed less interesting. Would be good to know.
-
Symmetry Reduces Parameters Through Parameter Tying
By
–
Symmetry can also exponentially reduce the number of parameters by parameter tying.
-
ConvNets Translation-Invariance and Layer Equivariance Properties
By
–
The output of a convnet is translation-invariant by design. Individual intermediate layers are equivariant. (Transformers may or may not be invariant/equivariant, btw.)
-
Full Distribution Access Versus Limited Sample Data
By
–
Ignore "infinite". What we really mean is having access to the the full distribution rather than just a sample.