More pre-training doesn't necessarily mean better generalization? This new paper discovers mode-hopping, where LLMs repeatedly switch between shallow pattern-matching and actual generalization, even while training loss remains stable. For example, OLMo3-32B went from 81%
Study Finds Language Models Switch Between Pattern Matching and Generalization
By
–
