In era of pretraining, what mattered was internet text. You'd primarily want a large, diverse, high quality collection of internet documents to learn from.

In era of supervised finetuning, it was conversations. Contract workers are hired to create answers for questions, a bit… https://t.co/rR6yYZGgKP

— Andrej Karpathy (@karpathy) 27 août 2025

In era of pretraining, what mattered was internet text. You'd primarily want a large, diverse, high quality collection of internet documents to learn from. In era of supervised finetuning, it was conversations. Contract workers are hired to create answers for questions, a bit

→ View original post on X — @karpathy,