Excellent playbook on the intricacies of distributed training. Covers basics of LLM training infra from the very ground-up, practicals/code, and efficiency benchmarks grounded on 4100 distributed experiments. Topics covered include:
– Data parallelism
– Tensor parallelism
– More
COMPUTING
-
Distributed Training Playbook: LLM Infrastructure Fundamentals and Benchmarks
By
–
-
TorchInductor Dynamic Kernel Generation Advances Tensor Compilation
By
–
TorchInductor supports dynamic kernel generation, and is likely the state of the art at symbolic Tensor type systems. long gone are the days of static shape tensor compilers
-
AI & Machine Learning Model for House Price Prediction
By
–
3. AI & Machine Learning Model Development
— God of Prompt (@godofprompt) 23 février 2025
Prompt:
"Develop a machine learning model in Python that predicts house prices based on location, square footage, and amenities. Use Pandas, Scikit-Learn, and Matplotlib to train, test, and visualize the results." pic.twitter.com/bvtSalnioP3. AI & Machine Learning Model Development Prompt: "Develop a machine learning model in Python that predicts house prices based on location, square footage, and amenities. Use Pandas, Scikit-Learn, and Matplotlib to train, test, and visualize the results."
-
CUDA through PyTorch sufficient for most training scenarios
By
–
Yes, I mean 99% of the time it's fine to just use CUDA through PyTorch (eager or compiled). If you train million-dollar-expensive LLMs, then maybe writing your own optimized CUDA kernels and custom NCCL would probably worthwhile so you can shave off some $$$ off your training
-
CUDA: NVIDIA GPU C++ Extension Behind PyTorch
By
–
CUDA is basically a C++ extension for NVIDIA GPUs. Basically what PyTorch uses under the hood.
-
Hello World: The Iconic First Program Turns 46
By
–
This month, “hello world” said “hello world!” The term was coined in a seminal programming book published in 1978: “C Programming Language,” written by Brian Kernighan and Dennis Ritchie: https://
go.aws/37TmOZq -
Wafer-Scale Processors: Yield, Defect Tolerance Solutions
By
–
🟧 How do we achieve a usable yield with a wafer-scale processor?
— Cerebras (@cerebras) 21 février 2025
❓ How good is our defect tolerance vs. GPUs?
💡 How do we manage defect tolerance?
Sean Lie breaks down the answers for us. 📽️ ⬇️ pic.twitter.com/K33UZTSzXDHow do we achieve a usable yield with a wafer-scale processor? How good is our defect tolerance vs. GPUs? How do we manage defect tolerance? Sean Lie breaks down the answers for us.
-
MLGym: Framework and Benchmark for AI Research Agents
By
–
MLGym: A New Framework and Benchmark for Advancing AI Research Agents Nathani et al.: https://
arxiv.org/abs/2502.14499 #ArtificialIntelligence #DeepLearning #Machinelearning -
CUDA Programming: Mindset Shift for Thread-Based Development
By
–
Fortunately, it's C-inspired. And I kind of know some CUDA basics but yeah, it's a bit of a mindset shift when working with threads I guess.
-
Considering Learning CUDA After 14 Years
By
–
CUDA has been on my to-learn list since 2011.
(Worked in a molecular dynamics lab that used it, but I never really learned it.)
Maybe it's finally time to take the plunge. (But then, I also saw the AI CUDA Engineer announcement yesterday….)