Fascinating underexplored scaling direction "Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima" This paper claims that pretraining loss is not a sufficient proxy for downstream capability. Since LLMs trained on mixed domains can reach the same
Nexus: Improving LLM Generalization Beyond Pretraining Loss
By
–
