andrej, your videos are the AI equivalent of a popstar releasing a new single
@alexandr_wang
-
AI’s Future Direction: Researcher Perspectives on Key Questions
By
–
7/ the answer to these questions is at the heart of where AI goes over the next few years. individual researchers' positions on these topics vary wildly (from my own polling), so the future is uncertain. regardless, AI will be one of the greatest projects in human history!
-
GPU Supply Distribution Among Major AI Labs
By
–
5/ there is another factor—which is that the $100B is spread between multiple labs. the largest labs (Google, OpenAI, Meta, Anthropic, etc.) may only have 10-20% of the total GPU supply from NVDA (sidenote: Google is the hardest to track because they use TPUs as well as GPUs)
-
GPU spending scale needed for hundred billion dollar model training
By
–
6/ so we won't see what would happen if you spent $100B of GPUs on a model for a while because of how spread out the GPU distributions are. we'd likely need to see $500B of aggregate GPU spend to have an individual model trained on $100B of GPUs (wow!)
-
Data abundance and algorithmic advances beyond current scaling laws
By
–
7/ in the "data wall will slow AI" case, progress will depend on: – methods for data abundance (above)
– algorithmic advances that advance us beyond the current scaling law paradigm -
Data Wall: AI Progress Bottleneck or Engineering Challenge?
By
–
6/ if you zoomed all the way out, the data wall will either look like: A) a short-term impediment that we will engineer our way out of
B) a very meaningful plateau to AI progress the industry is pretty split on which of these it is—there's no obvious answer -
Scaling AI: Expert Data, RL Environments, and Synthetic Data Methods
By
–
4/ there's a lot of arguments on both ends, but fundamentally scaling relies on expanding beyond existing internet data we need a number of methods for data abundance:
– high-end expert frontier data, which is more valuable than internet data
– RL environments
– synthetic data -
Scaling Laws: Data and Compute Trade-offs in AI Training
By
–
3/ the original scaling laws require a scaling of data alongside compute, and while you can still improve loss with more compute, it is much less efficient than if you scaled data as well
-
Compute vs Data: Two Schools of Thought on AI Progress
By
–
2/ there are 2 schools of thought: 1) compute is the only real bottleneck to AI progress. the more we spend, the closer we get to AGI 2) we are hitting a data wall which will slow progress regardless of how much compute we have
-
100B GPU Investment: Will Next-Gen AI Models Justify NVIDIA Spending?
By
–
1/ one of the biggest questions in AI today is: since GPT-4 was trained in fall 2022, we've collectively spent ~$100B on NVIDIA GPUs will the next generation of AI models' capabilities live up to that aggregate investment level? NVIDIA qtrly datacenter rev, by @Thomas_Woodside