DDP or FSDP2, depending on the type of model. If it can saturate compute with DDP, then that's simpler and already good. If not, FSDP2.
@soumithchintala
-
Optimizing AI Models with torch.compile and FSDP2
By
–
we spend a lot of time optimizing decently-sized models too.
For anything that is normal-sized (one node, 8 GPUs), torch.compile + FSDP2 should be largely sufficient to get to really good performance.
We've spent hundreds (or probably thousands) of programmer-years working on -
Async Re-explored in AI Industry Research
By
–
yes, async is actively being re-explored in the industry.
Arthur / M'aurelio have been publishing on it, so are some other folks. -
RAS Blackwell GPU Isolation and Host Boot Issues
By
–
RAS is blackwell+, so TBD. no single GPU isolation, whole host gets booted.
-
Scheduler Complexity and Infrastructure Provisioning Strategies
By
–
the scheduler side of things is a whole another level of complexity. we dont reprovision from scratch though.
-
Training Machine Learning Models on 10k H100 GPUs
By
–
"How to train a model on 10k H100 GPUs?"
has now been immortalized on my blog: https://
soumith.ch/blog/2024-10-0
2-training-10k-scale.md.html
… -
Targeting checkpointing improvements planned for machine learning
By
–
it's on our list of things to do (more targeting checkpointing, not FSDP). I wrote up a plan / doc early last year about it.
-
HBM Memory in Switches: Critical Infrastructure for AI Scale
By
–
one more thing to add.
at this scale, we also have to adjust the actual packet routing algorithms in our switches and NICs, to be able to load-balance well. Did you know switches have to have significant HBM memory as well (not just GPUs) because as the packets queue up, they -
Training Models at Scale: 10K+ H100 Infrastructure Guide
By
–
quick mini-post I wrote for @francoisfleuret broadly summarizing the things one needs to do to train models on 10k+ H100s.
-
Scaling AI Training Across Thousands of H100 GPUs
By
–
There's three parts. 1. Fitting as large of a network and as large of a batch-size as possible onto the 10k/100k/1m H100s — parallelizing and using memory-saving tricks.
2. Communicating state between these GPUs as quickly as possible
3. Recovering from failures (hardware,