"Bad Data, Good Models?" A surprising take on LLM pretraining. This paper flips the script: pretraining with more toxic data can actually improve post-training control. Using Olmo-1B variants, the authors show that toxicity becomes more linearly separable—making
Toxic Data Improves LLM Post-Training Control and Separability
By
–
