9/ If we can solve the supply chain issues, the future for our models is bright. We could expect continued progress with each successive generation for many years to come.
@alexandr_wang
-
GPU Supply Chain Bottlenecks: Chips, Power, Interconnection Challenges
By
–
6/ On compute—we have seen the astronomical investment into GPUs with NVIDIA's revenue growth. Staggering. Now the bottleneck is going to be—
– how many more chips can we make
– where will they go
– where will the power come from
– how tightly can we interconnect them -
Data Scarcity and Building Data Abundance for AI
By
–
7/ On data, it's a bit more complicated. We've exhausted most of the internet as pre-training data, and as previously mentioned, the gains are coming from post-training. The next 4 years will need us to overcome data scarcity and build data abundance.
-
Building Data Infrastructure for AI Progress
By
–
8/ This too will be one of the great infrastructure projects of our time—ensuring we build the means of production of new data across the frontier to fuel AI progress.
-
GPT-3 Marked Inflection Point for Language Models AI
By
–
3/ @scale_AI had started working on language models the year before on the very first RLHF experiments on GPT-2. But GPT-2 still felt very much like a toy at that time. GPT-3 was the first moment where it was obvious this would be the major theme of AI
-
OpenAI’s 2020 Scaling Laws Paper: Foundation of Modern AI
By
–
4/ The key insight of scaling laws was actually published earlier that year in Jan 2020 by OpenAI—"Scaling Laws for Neural Language Models" This paper and GPT-3 ultimately was the start of the current AI era of rapidly scaling language models—especially the compute + data needed
-
Scaling Compute and Data: The Infrastructure Challenge Ahead
By
–
5/ The next 4 years, like the last 4 years, will be all about scaling. Much of progress depends on our ability to keep scaling compute + data exponentially. This is no easy feat. Scaling data + compute will be some of the largest infrastructure projects of our time.
-
GPT-3 Breakthrough: Scaling Language Models Revealed AI Potential
By
–
2/ GPT-3 was when it first became clear what the potential of scaling language models was. The efficacy of GPT-3 took the AI community by surprise for the most part—the capabilities were staggering compared to everything that came before in NLP.
-
GPT-3 Anniversary: Four Years of Language Model Evolution
By
–
1/ Today is the 4th anniversary of the original GPT-3 paper—"Language Models are Few-Shot Learners" Some reflections on how the last 4 years have played out, and thoughts about the next 4 years