Also note that other threads have incorrectly stated that 100T is the amount of the data the models were trained on. According to Hyro AI, GPT-3 was trained on 570GB of data. And according to Sigmoid, GPT-2 was trained on 40GB. I have not seen a rumored amount of data for GPT-4.
Correcting misconceptions about GPT model training data amounts
By
–