In 2019, OpenAI announced GPT-2 with this post: https://
openai.com/index/better-l
anguage-models/
… Today (~5 years later) you can train your own for ~$672, running on one 8XH100 GPU node for 24 hours. Our latest llm.c post gives the walkthrough in some detail: https://
github.com/karpathy/llm.c
/discussions/677
… Incredibly, the
@karpathy
-
GPT-2 Training Now Affordable: $672 on Single GPU Node
By
–
-
LLM Assistance Significantly Improves Programming Workflow
By
–
Strong agree. LLM assist has changed and improved programming quite substantially for me. And there's still so much low-hanging fruit. I'd be quite price inelastic for a premium product.
-
pipeline() Becomes Turing Complete with Programmable kwargs
By
–
pipeline() will soon be Turing Complete, programmable by kwargs
-
Scalar-based Backpropagation: Foundation for Tensor Derivatives
By
–
“turned out that by only defining the derivatives for scalar values, it was sufficient to generalise to any higher dimensional Tensors. Therefore, I think building backpropagation intuition from the scalar valued perspective is extremely educational” Yep exactly. I think matrix
-
Native Speech-to-Speech Model Creates Pressure with Real-Time Interruption
By
–
Very close to my own experience earlier today talking to @kyutai_labs It’s just a lot of pressure 😀
— Andrej Karpathy (@karpathy) 4 juillet 2024
This is native speech to speech model like GPT4o that was demo’d (but not yet released). So it can hear and speak direct and you can interrupt it. But it can interrupt you, too 😅 https://t.co/SU16wG1GWHVery close to my own experience earlier today talking to @kyutai_labs It’s just a lot of pressure 😀
This is native speech to speech model like GPT4o that was demo’d (but not yet released). So it can hear and speak direct and you can interrupt it. But it can interrupt you, too -
Karpathy demonstrates copying prompt to Claude for video generation
By
–
I used it! (And by that I mean I copy pasted it to Claude.) Example:
— Andrej Karpathy (@karpathy) 4 juillet 2024
Slow panning shot: A Pride and Prejudice scene unfolds at a grand Regency-era manor. The five Bennet sisters, dressed in ornate 19th-century gowns, stand in a manicured garden. A wealthy, eligible bachelor… pic.twitter.com/dUsBVS9fqfI used it! (And by that I mean I copy pasted it to Claude.) Example: Slow panning shot: A Pride and Prejudice scene unfolds at a grand Regency-era manor. The five Bennet sisters, dressed in ornate 19th-century gowns, stand in a manicured garden. A wealthy, eligible bachelor
-
Struggles with video generation despite promising consistency results
By
–
I'm trying! People seem to be getting really good results with it but I can't quite get that myself so far. It's kind of ignoring my instructions and generating videos that look way too modern, or just wrong or unrelated. I'll keep trying because the consistency is really great.
-
AI Challenge Excitement and Enthusiasm
By
–
haha hey that sounds great, we want a real challenge for the AI 🙂
-
Creating Visual Stories with Generative AI Tools and Claude
By
–
I'm playing around with generative AI tools and stitching them together into visual stories. Here I took the first few sentences of Pride and Prejudice and made it into a video.
— Andrej Karpathy (@karpathy) 4 juillet 2024
The gen stack used for this one:
– @AnthropicAI Claude took the first chapter, generated the scenes… pic.twitter.com/vX64avfRUYI'm playing around with generative AI tools and stitching them together into visual stories. Here I took the first few sentences of Pride and Prejudice and made it into a video. The gen stack used for this one:
– @AnthropicAI Claude took the first chapter, generated the scenes -
State of the art image generation progress over seven years
By
–
I feel like I have to once again pull out this figure. These 32×32 texture patches were state of the art image generation in 2017 (7 years ago). What does it look like for Gen-3 and friends to look similarly silly 7 years from now. https://t.co/MOXtfYahIW pic.twitter.com/tv4DWZjWCu
— Andrej Karpathy (@karpathy) 1 juillet 2024I feel like I have to once again pull out this figure. These 32×32 texture patches were state of the art image generation in 2017 (7 years ago). What does it look like for Gen-3 and friends to look similarly silly 7 years from now.