"Self-Play Pretraining with Zero Data" Most pretraining still depends on humans deciding what data a model should learn from, so can a model learn entirely through self-play and no real data? To test that, they trains two models from scratch. One writes small programs that
Study Tests Language Model Pretraining Through Self-Play Without Real Data
By
–
