I discussed the challenges of silent data corruption in ML training jobs, and how one faulty piece of hardware can infiltrate and affect the results of a large scale training jobs on thousands of chips.
MACHINE LEARNING
-
Gemini Model Training: Challenges and Recent Advances
By
–
I gave an overview of some of the challenges and advances in our recent Gemini model training runs.
-
ML Training CO2 Emissions: Debunking Highly Cited Misinformation
By
–
I discussed misinformation in the area of CO2 emissions from ML training, where one highly cited paper has a misleading and incorrect estimate that is off by 88X from the actual CO2 emissions of a one-time neural architecture search done by So et al., and is off by 118,000X from
-
Reinforcement Learning Advances TPU Chip Design Across Generations
By
–
Among many other topics, here are a few things from the talk. I gave an update on our use of reinforcement learning for ASIC chip placement and routing, showing how the adoption of this approach has been increasing from TPU generation to generation (due to further work on the
-
Impressive Test Results From Recent AI Development Work
By
–
Ugh been so busy w other stuff haven’t built much recently but the few tests I ran were really impressive.
-
MLSys Talk on ML for Systems and Systems for ML
By
–
I had a great time giving a talk at MLSys this morning on "Advances in ML for Systems and Systems for ML". So many enthusiastic questioners! https://
mlsys.org -
BAIR Students and Alumni Win Major Awards at ICRA2024
By
–
Hearty congratulations to BAIR students, faculty and alumni for their many awards at #ICRA2024 this week in Japan BAIR alumni @pulkitology @LerrelPinto @RCalandra won Early Career awards; students from @svlevine @Ken_Goldberg @JitendraMalikCV labs won both Best Paper awards!
-
Workshop on Multi-Object Grasping at ICRA 2024
By
–
Also looking forward to Workshop on Multi-Object Grasping: Progress and Prospects today #ICRA2024 in CC-313:
-
GPT model MMLU performance shows slowing progress
By
–
GPT-3’s score on the MMLU benchmark was 40%. First release of GPT-4 scored 86%, and today GPT-4o is 89%. An increase of just 3% — that’s a full year of progress. If you plot the prior trend we were supposed to be at 100, maybe 120% by now. AI is hitting a wall.
-
AlphaGo Tuning with Bayesian Optimization Improved Win Rate
By
–
I agree with this thread of @avt_im
. Prior to the match with Lee Sedol, we tuned the latest AlphaGo agent with Bayesian Optimization and this improved its win-rate from 50% to 66.5% in self-play games. This tuned version was deployed in the final match. See