Excellent playbook on the intricacies of distributed training. Covers basics of LLM training infra from the very ground-up, practicals/code, and efficiency benchmarks grounded on 4100 distributed experiments. Topics covered include:
– Data parallelism
– Tensor parallelism
– More
Distributed Training Playbook: LLM Infrastructure Fundamentals and Benchmarks
By
–
