two or three years ago, this was a prevailing sentiment “training the largest LLMs is very hard. only a few people know how. everyone else is failing” it was hard to avoid huge loss spikes. now pretraining is a solved problem. what changed? do we just clean our data better?

→ View original post on X — @jxmnop,