It was quite obvious 5-10 years ago that learning the structure of language by predicting missing words was easier than learning, say, intuitive physics by predicting video.
We have had projects on both things at FAIR for almost 10 years, and the language side made progress much

→ View original post on X — @ylecun,