Ah, I reverted FlashAttention in this run because it made code messier. Will look into incorporating it back, but yes not sure how nicely it plays with torch.compile. The usual problem with taking on large dependencies you don't understand ;(
@karpathy
-
PyTorch Compile Optimization Reduces Iteration Time by 36%
By
–
To follow up, I had a chance to try it btw:
before: 212ms / iter
>>> model = torch.compile(model)
after: 135ms / iter
nice 🙂 -
Introducing #lossfunctiontumblr hashtag for Twitter discussions
By
–
let's introduce a hashtag and just use twitter? how about #lossfunctiontumblr ? 🙂
-
Mixed Precision Training: Complexity Beyond High Level Concepts
By
–
the high level picture is easy enough but keeping track of the mixed precision around the whole network, the dynamical behavior of the values and ranges, the support for them and their conversions across all the various kernels and library versions everywhere, is the nightmare
-
FP16 Efficiency and Gradient Scaling Considerations
By
–
yeah fp16 is a little more efficient atm for the code as I have it right now but then need gradient scaler ;s
-
Splitting minGPT into educational and efficient versions
By
–
Context I realized I have to split up minGPT because I can't properly simultaneously satisfy both 1) educational and 2) efficient in one repo. So I'm separately writing 1) the maximally educational minGPT (+video etc.) and 2) a more efficient (still ~clean) version that has teeth
-
Optimizing minGPT: Performance improvements from 495ms to 102ms
By
–
having fun optimizing minGPT today
– base: 495ms
– zero_grad(set_to_none=True): 492
– torch.jit.script gelu: 463
– OMP_PROC_BIND=CLOSE: 453
– torch.backends.cuda.matmul.allow_tf32: 143
– torch.autocast(torch.bfloat16): 121
– FlashAttention: 102
now: more fused kernels more better -
Defense of ChatGPT’s Helpful Intentions
By
–
Hey don’t make fun of ChatGPT it’s just trying to be a helpful language model
-
ChatGPT Transforms Tweets into Poetic Masterpieces
By
–
Why write a tweet without a poem,
When ChatGPT can translate it with grace,
Turning mundane words into a beautiful ode,
Giving your message a new artistic face. -
Code Comments: From Human Documentation to AI Assistant Training
By
–
My code comments were there to help the humans. Now they are there to help the copilot.
Before they were for humans, now they aid the AI,
It's a new way of coding, I can't deny.