This is a brilliant explanation of why next word prediction is a good loss function to achieve understanding. In addition to predicting the next bit, one can also use masking losses and diffusion to achieve understanding. The general intuition is to minimise the difference… https://t.co/tUDUQWP4NJ
— Nando de Freitas (@NandoDF) 25 mai 2025
This is a brilliant explanation of why next word prediction is a good loss function to achieve understanding. In addition to predicting the next bit, one can also use masking losses and diffusion to achieve understanding. The general intuition is to minimise the difference