Check out DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining, and some of the papers it cites. By @sangmichaelxie et al.

→ View original post on X — @hardmaru