I think it only works when you have a larger model produce data for a smaller one. Otherwise the large model -> same large model, without any other external signals, will likely just learn something very similar
MACHINE LEARNING
-
KL Divergence: The Foundation for Modern Language Models
By
–
http://
blog.alexalemi.com KL is All You Need https://
bit.ly/3Ll57rQ
#AI #MachineLearning #DeepLearning #LLMs #DataScience -
Google’s NeuralGCM AI Model Advances Climate Prediction Accuracy
By
–
Exciting news! Meet NeuralGCM, a cutting-edge atmospheric model from Google Research. 🌡🌎
— Jeff Dean (@JeffDean) 22 juillet 2024
This AI-driven general circulation model (GCM) outperforms traditional atmospheric models in precision, providing faster, more accurate climate predictions. Learn more in @Nature →… https://t.co/Ej9x1HIN4CExciting news! Meet NeuralGCM, a cutting-edge atmospheric model from Google Research. This AI-driven general circulation model (GCM) outperforms traditional atmospheric models in precision, providing faster, more accurate climate predictions. Learn more in @Nature →
-
Google NeuralGCM: Hybrid AI Solver for Weather Modeling
By
–
Research Engineer Dmitrii Kochkov's work on weather & climate contributed to the development of a hybrid AI + differentiable solver approach for atmospheric modeling.
— Google AI (@GoogleAI) 22 juillet 2024
Learn more about the groundbreaking work from Google Research's NeuralGCM team. 🌡⛈️→ https://t.co/QJ6cusDKZh pic.twitter.com/1ZiGduq0jMResearch Engineer Dmitrii Kochkov's work on weather & climate contributed to the development of a hybrid AI + differentiable solver approach for atmospheric modeling. Learn more about the groundbreaking work from Google Research's NeuralGCM team. → http://
goo.gle/4ff57ah -
Iteration in AI: Small Steps, Big Results for Development
By
–
✨ Iteration in AI: Small steps, big results! Dive into our #AIToday episode to understand the impact of iterative processes on AI development. 🎧 cognilytica.com/what-iterati… #AIProjects #Innovation #Podcast
-
Visual Haystacks: Advanced Computer Vision Search Techniques
By
–
New blog post: https://
bair.berkeley.edu/blog/2024/07/2
0/visual-haystacks/
… -
The Danger Zone in Data Science
By
–
The Danger Zone in Data Science https://
bit.ly/4cVUNCo
#AI #MachineLearning #DeepLearning #LLMs #DataScience -
Small Models Large Context Windows: The Winning LLM Formula
By
–
The winning formula for LLMs is going to be (a) small model, (b) large context window, (c) strong reasoning ability. The quality cost trade off makes this clear: https://
x.com/swyx/status/18
15037679014388172
… Large context window will enable a lot of applications (prompt instructions, RAG) as long -
Open Source to Dominate: Smaller Models Over Large Language Models
By
–
This also means that Open Source is going to dominate in the future – training and serving smaller models is more accessible to many. The large models are going to useful for generating and filtering data, while the smaller models (e.g., 4o-mini) are the ones we want deployed.
-
GPT-2 Sized Model Demonstrates Universal Physical Understanding
By
–
Building AI with universal physical understanding will have a tremendous impact. We have taken the first steps towards this by building a ~GPT-2 sized model that shows that a broad model can capture a wide range of physical phenomena. https://
arxiv.org/pdf/2403.03542 We have scaled the