100% agree, "the proof is in the pudding". It has to actually work. I will say that I think the technology exists today to actually make it work at the needed threshold. Actually making it work is still difficult. But 6 years ago I would have said the technology does not exist.
@karpathy
-
Apple Intelligence: Multimodal AI Integration Across iOS
By
–
Actually, really liked the Apple Intelligence announcement. It must be a very exciting time at Apple as they layer AI on top of the entire OS. A few of the major themes. Step 1 Multimodal I/O. Enable text/audio/image/video capability, both read and write. These are the native
-
Icon Color Customization Before AGI Deployment
By
–
I am also exhilarated to learn that you can now change the color of your icons, and that you can choose ANY color you want, right before we look at how we deploy SOTA AGI to a few billion devices.
-
Apple’s AI announcements at WWDC 2024 underwhelm expectations
By
–
If you tuned in to WWDC to see what Apple is doing with AI, we're all probably thinking the same thing around now 50 minutes into it…
-
GPT-2 Speed Run: Building 124M Model from Scratch
By
–
"GPT-2 speed run" haha I love that.
From empty file to GPT-2 (124M) 😀 -
llm.c optimization: balancing historical accuracy and training speed
By
–
This is cool!! I'm not exactly sure how to upstream these changes to llm.c… Part of me wants to reproduce GPT-2/3 using their exact hyperparameters just for historical aesthetics, but part of me also wants to just train things as fast as possible. Probably both. – lr 3X is
-
Learning Rate Annealing Schedules: Examples and Best Practices
By
–
What is larger or higher here, do you have example annealing schedule you’ve found worked well?
-
Bug Fix: Gradient Norm Clipping Synchronization Issue with ZeRO-1
By
–
There was a bug with gradient norm clipping, it was incorrectly synchronized across GPUs when using ZeRO-1, which may have contributed to the loss spike I saw earlier. It's fixed now on master. More generally as of yesterday though we've re-established full and exact equality to
-
Question about Flash Attention in cuDNN Implementation
By
–
Sorry can you expand? Are you saying this might be due to the flash attention inside cuDNN?