Great question yes I was surprised that 10B seemed enough. I believe GPT-2 was trained on somewhere ~100B tokens. The reason we reach this performance in 10B tokens I think may be the following: 1. FineWeb could just be higher quality than WebText, on a per-token basis. This was
@karpathy
-
Start Latency Issues with Dataset Tokenization in Python
By
–
TIL, will look into! The thing that makes this a bit complicated right now is the start latency. What bloats up the setup time right now is the dataset and its tokenization, which is all done in Python right now. Installing huggingface datasets, downloading FineWeb 10B and
-
H100 GPU Shortage Challenges Training Cost Reduction
By
–
Yep, moving to H100 is one easy way to bring down the estimates here. Sadly I can't find any H100 GPUs
-
Training Loss Noise and Validation Loss Smoothing Explained
By
–
Training loss is evaluated over the batch, i.e. 0.5M tokens. It's noisy but this is expected, you could be iterating through easy or hard documents in the training data. The validation loss is averaged over 20 batches of 0.5M tokens (this is a hyperparameter), so it is smoother.
-
Conservative hyperparameters with potential for 10X speedup
By
–
Agree!! I'm using very conservative settings for a lot of the hyperparameters (following GPT-3 paper when possible) and haven't tried to speed this up at all yet, but I expect a 10X multiplier here should be possible.
-
Reproduce GPT-2 124M in llm.c for $20 in 90 Minutes
By
–
# Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 The GPT-2 (124M) is the smallest model in the GPT-2 series released by OpenAI in 2019, and is actually quite accessible today, even for the GPU poor. For example, with llm.c you can now reproduce this model on one 8X
-
Transformer blocks progressively refine information through attention mechanism
By
–
(More likely though, each block refines the information over time in the Transformer forward pass, enriching it with the information gathered from previous tokens during Attention.)
-
Residual Stream Projections and Information Preservation in Neural Networks
By
–
Of course it has access, the projections from each block into the residual stream can be learned to be zero and so preserve any information that is needed.
-
Modern AI Capabilities Compared to Early Computing Hardware Specs
By
–
Difficult to not feel like it is the equivalent of something along the lines of "CPU with a clock rate of more than 10^7 Hz and 20MiB RAM".
-
Toaster storytelling vision for morning routines
By
–
We cannot rest until the toaster tells a tiny story while waiting for the bread in the morning 😀