Do you work with audio data? If we add a Keras layer to convert raw audio to Mel spectrograms, would you be interested?
@fchollet
-
AGI on the Horizon: Yet Bot Detection Remains Challenging
By
–
Yes, we may be on the cusp of AGI. But using AI to automatically detect and block bots like this remains elusive, due to their use of a remarkably sophisticated evasion technique — whitespace.
-
XOR Representability vs Efficiency in Program Search
By
–
There's no connection, XOR was a representability issue. It's easy to fit a curve to a XOR dataset. That said, doing so is tremendously inefficient compared to program search over a DSL that just has NOR, for instance. So if you want a parallel, it's there.
-
Transformers Excel at Interpolation but Struggle with Symbolic Learning
By
–
Not saying that Transformers are worse than RNNs, mind you — Transformers are *the best* at *what deep learning does* (generalizing via interpolation), specifically *because* of their strongly interpolative architecture prior (MHA). They are, however, worse at learning symbolic
-
RNNs: The Core Principle of Recurrence and Latent Space Iteration
By
–
RNNs are… recurrent. It's right there in the name. That was the entire intuition behind creating them: iterating in latent space, like the brain.
-
Transformers’ Interpolative Architecture and Limitations for Symbolic Tasks
By
–
Ironically, Transformers are even worse in that regard — mostly due to their strongly interpolative architecture prior. Multi-head-attention literally hardcodes sample interpolation in latent space. Also, the fact that recurrence is a really helpful prior for symbolic programs.
-
RNNs and Universal Computation: A 2013-2016 Perspective
By
–
From 2013 to 2016 I was actually quite convinced that RNNs could be trained to learn any program. After all, they're Turing-complete (or at least some of them are) and they learn a highly compressed model of the input:output mapping they're trained on (rather than mere pointwise
-
The Limitations of Curve-Fitting for Symbolic Reasoning Tasks
By
–
The "aha" moment when I realized that curve-fitting was the wrong paradigm for achieving generalizable modeling of problems spaces that involve symbolic reasoning was in early 2016. I was trying every possible way to get a LSTM/GRU based model to classify first-order logic
-
Generalization vs Overfitting: Understanding Model Performance Gaps
By
–
Lack of generalization != overfitting. Lack of generalization could come from overfitting, underfitting, concept drift, OOD, or could come up for a model where there's no "fitting" at all. It just means that the model cannot operate in situations it hasn't been prepared for
-
Limitations of Training AI Models on Synthetic and Video Data
By
–
But you can't just fit a big curve to a bunch of UE5 screencaps and YouTube videos and expect a generalizable model of the real world. That is not how these models work.