Excellent playbook on the intricacies of distributed training. Covers basics of LLM training infra from the very ground-up, practicals/code, and efficiency benchmarks grounded on 4100 distributed experiments. Topics covered include:
– Data parallelism
– Tensor parallelism
– More
HARDWARE
-
Distributed Training Playbook: LLM Infrastructure Fundamentals and Benchmarks
By
–
-
CUDA through PyTorch sufficient for most training scenarios
By
–
Yes, I mean 99% of the time it's fine to just use CUDA through PyTorch (eager or compiled). If you train million-dollar-expensive LLMs, then maybe writing your own optimized CUDA kernels and custom NCCL would probably worthwhile so you can shave off some $$$ off your training
-
CUDA: NVIDIA GPU C++ Extension Behind PyTorch
By
–
CUDA is basically a C++ extension for NVIDIA GPUs. Basically what PyTorch uses under the hood.
-
Wafer-Scale Processors: Yield, Defect Tolerance Solutions
By
–
🟧 How do we achieve a usable yield with a wafer-scale processor?
— Cerebras (@cerebras) 21 février 2025
❓ How good is our defect tolerance vs. GPUs?
💡 How do we manage defect tolerance?
Sean Lie breaks down the answers for us. 📽️ ⬇️ pic.twitter.com/K33UZTSzXDHow do we achieve a usable yield with a wafer-scale processor? How good is our defect tolerance vs. GPUs? How do we manage defect tolerance? Sean Lie breaks down the answers for us.
-
CUDA Programming: Mindset Shift for Thread-Based Development
By
–
Fortunately, it's C-inspired. And I kind of know some CUDA basics but yeah, it's a bit of a mindset shift when working with threads I guess.
-
Topological Qubits: Quantum Computing Breakthrough Explained
By
–
Perfect use case for Think Deeper: what's a topological qubit? https://t.co/4z6Lur3H7c pic.twitter.com/MkwSGI7sAe
— Mustafa Suleyman (@mustafasuleyman) 21 février 2025Perfect use case for Think Deeper: what's a topological qubit? https://
aka.ms/AAu76rr -
SambaNova Achieves 198 Tokens Per Second With Custom AI Chips
By
–
Take it from @techradar SambaNova runs @deepseek_ai R1 at 198 tokens/sec using 16 custom chips! Our SN40L RDU chip is 3X faster & 5X more efficient than GPUs 5X speed boost coming soon, with 100X capacity on cloud. https://
bit.ly/3Xatj6k -
DeepSeek-R1 Achieves 95% Chip Reduction Industry-First Performance
By
–
@techradar reported that SambaNova’s @deepseek_ai
-R1 achieved industry-first #AI performance with 95% fewer chips. @RodrigoLiang told @Waynewill1
: “We’re collapsing inference costs and slashing hardware demands—from 40 racks to just one.” #GPU #DeepSeek -
Microsoft’s topological qubit arrives seven years late
By
–
Microsoft’s ‘topological qubit’ is seven years late and a million bits short Microsoft told me they would have a “topological qubit” in 2018. It’s only just arrived this week — maybe. There’s skepticism among scientists as to whether Microsoft’s paper in Nature actually proves
-
Grok’s H100 GPU Training: Missing Transparency on Key Metrics
By
–
The number of H100 GPUs used for training helps estimate parameters, data size, and training duration, so it’s an important detail. But Grok’s team hasn’t been clear about key factors like benchmark performance and the exact number of GPUs used for training, which leads to