6/ so we won't see what would happen if you spent $100B of GPUs on a model for a while because of how spread out the GPU distributions are. we'd likely need to see $500B of aggregate GPU spend to have an individual model trained on $100B of GPUs (wow!)
RESEARCH
-
Data Wall: AI Progress Bottleneck or Engineering Challenge?
By
–
6/ if you zoomed all the way out, the data wall will either look like: A) a short-term impediment that we will engineer our way out of
B) a very meaningful plateau to AI progress the industry is pretty split on which of these it is—there's no obvious answer -
Data abundance and algorithmic advances beyond current scaling laws
By
–
7/ in the "data wall will slow AI" case, progress will depend on: – methods for data abundance (above)
– algorithmic advances that advance us beyond the current scaling law paradigm -
Scaling Laws: Data and Compute Trade-offs in AI Training
By
–
3/ the original scaling laws require a scaling of data alongside compute, and while you can still improve loss with more compute, it is much less efficient than if you scaled data as well
-
Scaling AI: Expert Data, RL Environments, and Synthetic Data Methods
By
–
4/ there's a lot of arguments on both ends, but fundamentally scaling relies on expanding beyond existing internet data we need a number of methods for data abundance:
– high-end expert frontier data, which is more valuable than internet data
– RL environments
– synthetic data -
Compute vs Data: Two Schools of Thought on AI Progress
By
–
2/ there are 2 schools of thought: 1) compute is the only real bottleneck to AI progress. the more we spend, the closer we get to AGI 2) we are hitting a data wall which will slow progress regardless of how much compute we have
-
Generative ConvNets for Video: A Decade of AI Innovation
By
–
Us indeed! A decade earlier, in 2010, we organically introduced generative convnets for video understanding and generation. I remember presenting it at NeurIPS and Jitendra Malik rightfully asking us to scale and evaluate. Yet, I argued against most of the vision community at the
-
LazyPredict: Train and Evaluate Multiple ML Models Instantly
By
–
4. LazyPredict Lazy Predict allows you to train, test and evaluate all your ML models at once with just a few lines of code.
-
7 Amazing Python Libraries for Data Science and Machine Learning
By
–
7 amazing Python libraries for Data Science and Machine Learning not known to many:
-
Video Generation Models Progress Beyond Current Limitations
By
–
The progress in video generation is astounding. Sora, Veo, Kling, etc are only the GPT2s of video at best. When the GPT4 equivalents arise, machines will be able to do reasoning with video, chain-of-thought with video, multimodal generalisation, inference and control from