I'm sorry, you're right, H100 not A100 => ~4X compute numbers.
@karpathy
-
Cost analysis of training 8B and 70B language models on A100
By
–
Napkin math here is 1 A100 hour atm is ~$1 on cloud providers, so roughly 1.3M hours for 8B (see model card) would mean $1.3M. And $6.4M for 70B. Keeping in mind that this is just the approx cost to hit go and wait and assuming a perfect run. And that it takes quite a bit more in
-
Llama 3 8B Performance Comparable to Llama 2 70B Model
By
–
The model card has some more interesting info too: https://
github.com/meta-llama/lla
ma3/blob/main/MODEL_CARD.md
… Note that Llama 3 8B is actually somewhere in the territory of Llama 2 70B, depending on where you look. This might seem confusing at first but note that the former was trained for 15T tokens, while the -
Chinchilla Scaling Laws: Compute Optimality vs Convergence Point
By
–
no. people misunderstand chinchilla.
chinchilla doesn't tell you the point of convergence.
it tells you the point of compute optimality.
if all you care about is perplexity, for every FLOPs compute budget, how big model on how many tokens should you train?
for reasons not fully -
Meta Releases Llama 3 Models in 8B and 70B Variants
By
–
Congrats to @AIatMeta on Llama 3 release!! https://
ai.meta.com/blog/meta-llam
a-3/
…
Notes: Releasing 8B and 70B (both base and finetuned) models, strong-performing in their model class (but we'll see when the rankings come in @ @lmsysorg :))
400B is still training, but already encroaching -
Dataset Limitations: Missing the Full Distribution of Correct Answers
By
–
Issue in mind is not so much human bias but the fact that the full distribution of correct or desirable answers to your prompts is almost certainly not present in your dataset, only a few samples.
-
The Ambiguity Problem in LLM Labeling and Training Data
By
–
Consider being a labeler for an LLM. The prompt is “give me a random number between 1 and 10”. What SFT & RM labels do you contribute? What does this do the network when trained on? In subtle way this problem is present in every prompt that does not have a single unique answer.
-
Five Years Between Transformer Attention and FlashAttention Innovation
By
–
"5 years between Self-Attention Is All You Need and FlashAttention"
quite incredible stat, gives a pause -
Modeling Different Coprocessors with Varying Instruction Sets
By
–
another layer of the onion 🙂 maybe can you model it like having different kinds of coprocessors with different instruction sets / reliability rates
-
Visualizing Computation DAGs: Identifying Narrow and Wide Bottlenecks
By
–
sorry i just made that word up in my head right now, didn't mean to hijack some existing term. I meant – imagine the computation as a DAG, lay it out in your head, and look for "narrow" and "wide" parts. Open to alternatives for future use!