Distillation has been on the news (!) due to @deepseek_ai
. The paper https://
arxiv.org/abs/1503.02531 was actually rejected from NeurIPS 2014 due to lack of novelty (true-ish), and lack of impact . Thanks reviewer#2 (literally), and thanks for @arxiv
! @geoffreyhinton @JeffDean
@oriolvinyalsml
-
Knowledge Distillation Gains Spotlight Through DeepSeek Research
By
–
-
Gemini 2.0 Pro Exp: Building 3D Solar System Animation
By
–
The Gemini 2.0 family grows! I love astronomy, so what better way to celebrate than to code up an animated 3D scene of the solar system with Gemini 2.0 Pro Exp 🪐🌞 Let’s keep the balls rolling! https://t.co/xcVMbcF5ig pic.twitter.com/8tFFA2ibTa
— Oriol Vinyals (@OriolVinyalsML) 5 février 2025The Gemini 2.0 family grows! I love astronomy, so what better way to celebrate than to code up an animated 3D scene of the solar system with Gemini 2.0 Pro Exp Let’s keep the balls rolling!
-
Thinking Models Predict Future Outcomes With Inference Accuracy
By
–
As it's a thinking model you can see why it concludes certain things (e.g. that I would go to industry vs academia), and the inferences are spot on. But then it actually verifies what happened as indeed, it "knows" the future.
-
Gemini Thinking Release Shows Rapid Progress
By
–
Great & super fast progress from our first Thinking release in December, kudos to @jack_w_rae @NoamShazeer and the whole Gemini team!
-
AI Predicts Career Path and Evaluates Academic Thesis
By
–
Quite funny to ask it to predict my future career (pretty accurately I must say!), or to roast the thesis (Q_Q).
-
PhD Thesis Evaluated by Flash Thinking Model for Theoretical Gaps
By
–
I uploaded my PhD thesis (98K tokens) and asked our latest Flash thinking model to find flaws in it. "Potential over-reliance on empirical results without sufficient theoretical grounding." A Deep Learning thesis, indeed.
-
Felix Tribute Visual Language Models Pioneer DeepMind
By
–
Very saddened by this. I was lucky to interact with Felix at DeepMind and his creative thinking was clearly ahead of his time. Our last work together: one of the very first Visual Language Models: https://
arxiv.org/abs/2106.13884 RIP Felix. -
Seq2seq Architecture: A Decade of Evolution in Deep Learning
By
–
seq2seq 10 years later. 💙@ilyasut @quocleix. Talk 👇 https://t.co/fqZI0SXdN0 pic.twitter.com/Q5le02u1fE
— Oriol Vinyals (@OriolVinyalsML) 14 décembre 2024seq2seq 10 years later. @ilyasut @quocleix
. Talk -
Curriculum Learning: Progressive Neural Network Training Strategy
By
–
We didn't investigate this very intensely but my intuition is that this acted like a curriculum, so that you could learn the "short" connections first, and start going backwards. C' B' A' A B C (learning A' to A translation is simple as there aren't many layers in between).