Cool! For the spike I'd try e.g. `-sl 7 -sg 7` to keep instability in check earlier in the training. (will skip update if loss/gradnorm > 7 sigma outlier is detected)
@karpathy
-
GPT-2: Perfect Introduction to LLMs and Distributed Training
By
–
Exactly as intended! GPT-2 is a beautiful "hello world" to LLMs but also distributed training etc.
-
Estimating compute improvements from 2019 to 2024
By
–
This information was never released but I'd expect it was a lot more. In terms of multipliers let's say 3X from data, 2X from hardware utilization, in 2019 this was probably a V100 cluster (~100 fp16 TFLOPS), down from H100 (~1,000), so that's ~10X. Very roughly let's say ~100X
-
Arithmetic Operation for Layer Norm Standard Deviation Calculation
By
–
Do you see an arithmetic operation that could help us calculate this layernorm standard deviation? pic.twitter.com/nV4wwJRCLB
— Andrej Karpathy (@karpathy) 11 juillet 2024Do you see an arithmetic operation that could help us calculate this layernorm standard deviation?
-
GPT-2 Training Now Affordable: $672 on Single GPU Node
By
–
In 2019, OpenAI announced GPT-2 with this post: https://
openai.com/index/better-l
anguage-models/
… Today (~5 years later) you can train your own for ~$672, running on one 8XH100 GPU node for 24 hours. Our latest llm.c post gives the walkthrough in some detail: https://
github.com/karpathy/llm.c
/discussions/677
… Incredibly, the -
LLM Assistance Significantly Improves Programming Workflow
By
–
Strong agree. LLM assist has changed and improved programming quite substantially for me. And there's still so much low-hanging fruit. I'd be quite price inelastic for a premium product.
-
pipeline() Becomes Turing Complete with Programmable kwargs
By
–
pipeline() will soon be Turing Complete, programmable by kwargs
-
Scalar-based Backpropagation: Foundation for Tensor Derivatives
By
–
“turned out that by only defining the derivatives for scalar values, it was sufficient to generalise to any higher dimensional Tensors. Therefore, I think building backpropagation intuition from the scalar valued perspective is extremely educational” Yep exactly. I think matrix
-
Native Speech-to-Speech Model Creates Pressure with Real-Time Interruption
By
–
Very close to my own experience earlier today talking to @kyutai_labs It’s just a lot of pressure 😀
— Andrej Karpathy (@karpathy) 4 juillet 2024
This is native speech to speech model like GPT4o that was demo’d (but not yet released). So it can hear and speak direct and you can interrupt it. But it can interrupt you, too 😅 https://t.co/SU16wG1GWHVery close to my own experience earlier today talking to @kyutai_labs It’s just a lot of pressure 😀
This is native speech to speech model like GPT4o that was demo’d (but not yet released). So it can hear and speak direct and you can interrupt it. But it can interrupt you, too -
Karpathy demonstrates copying prompt to Claude for video generation
By
–
I used it! (And by that I mean I copy pasted it to Claude.) Example:
— Andrej Karpathy (@karpathy) 4 juillet 2024
Slow panning shot: A Pride and Prejudice scene unfolds at a grand Regency-era manor. The five Bennet sisters, dressed in ornate 19th-century gowns, stand in a manicured garden. A wealthy, eligible bachelor… pic.twitter.com/dUsBVS9fqfI used it! (And by that I mean I copy pasted it to Claude.) Example: Slow panning shot: A Pride and Prejudice scene unfolds at a grand Regency-era manor. The five Bennet sisters, dressed in ornate 19th-century gowns, stand in a manicured garden. A wealthy, eligible bachelor