Ironically, Transformers are even worse in that regard — mostly due to their strongly interpolative architecture prior. Multi-head-attention literally hardcodes sample interpolation in latent space. Also, the fact that recurrence is a really helpful prior for symbolic programs.
MACHINE LEARNING
-
RNNs and Universal Computation: A 2013-2016 Perspective
By
–
From 2013 to 2016 I was actually quite convinced that RNNs could be trained to learn any program. After all, they're Turing-complete (or at least some of them are) and they learn a highly compressed model of the input:output mapping they're trained on (rather than mere pointwise
-
The Limitations of Curve-Fitting for Symbolic Reasoning Tasks
By
–
The "aha" moment when I realized that curve-fitting was the wrong paradigm for achieving generalizable modeling of problems spaces that involve symbolic reasoning was in early 2016. I was trying every possible way to get a LSTM/GRU based model to classify first-order logic
-
Sora will generate multiple videos side-by-side simultaneously
By
–
Sora will be able to generate multiple videos side-by-side simultaneously.
— AI Breakfast (@AiBreakfast) 17 février 2024
This is a single video sample from Sora, from OpenAI developer @billpeeb
pic.twitter.com/6CKQb7LSaGSora will be able to generate multiple videos side-by-side simultaneously. This is a single video sample from Sora, from OpenAI developer @billpeeb
-
Medical AI Innovations Help Burn Victims Worldwide
By
–
Medical Innovations, Vast Experience Combine To Help Burn Victims In Israel And Worldwide https://
forbes.com/sites/gilpress
/2024/01/21/medical-innovations-vast-experience-combine-to-help-burn-victims-in-israel-and-worldwide/?utm_source=dlvr.it&utm_medium=twitter
… #Innovation #ArtificialIntelligence #MachineLearning -
Gary’s AI Skepticism Shifts from Limitations to Safety Concerns
By
–
Gary always hates AI, but notice how he has gone from "end to end learning will never work" and "deep learning has hit a wall" to "we are so screwed because now it works too well"
-
3D animation generated from 2D video by MoveAI enhances AR/VR
By
–
3D animation generated from a standard 2D video using @MoveAI_
— AI Breakfast (@AiBreakfast) 17 février 2024
Significant leap for AR/VR tech.
pic.twitter.com/MjBBIhow3y3D animation generated from a standard 2D video using @MoveAI_ Significant leap for AR/VR tech.
-
Generalization vs Overfitting: Understanding Model Performance Gaps
By
–
Lack of generalization != overfitting. Lack of generalization could come from overfitting, underfitting, concept drift, OOD, or could come up for a model where there's no "fitting" at all. It just means that the model cannot operate in situations it hasn't been prepared for
-
nanoGPT with MoEs and multimodal tokenization support
By
–
i would love for @karpathy to make a nanoGPT for MoEs with multimodal tokenization support
-
Limitations of Training AI Models on Synthetic and Video Data
By
–
But you can't just fit a big curve to a bunch of UE5 screencaps and YouTube videos and expect a generalizable model of the real world. That is not how these models work.