Amazing also check out all ECCV papers here:
RESEARCH
-
Training Machine Learning Models on 10k H100 GPUs
By
–
"How to train a model on 10k H100 GPUs?"
has now been immortalized on my blog: https://
soumith.ch/blog/2024-10-0
2-training-10k-scale.md.html
… -
Historical Materialism Limitations and Psychoanalytic Theory
By
–
This is where historical materialism fails us and we need ~Freud
-
Targeting checkpointing improvements planned for machine learning
By
–
it's on our list of things to do (more targeting checkpointing, not FSDP). I wrote up a plan / doc early last year about it.
-
Larry Roberts ARPANet 1967 networking breakthrough
By
–
#otd in 1967 Larry Roberts presented his idea for an “ARPANet” for connecting multiple computers together across the United States. Full paper: https://
bit.ly/3TLjqKP -
Data Engineers Critical for GenAI Success and Code Development
By
–
The best #GenAI models will not succeed without good data. That’s why data engineers are even more critical today. This guide shares practical guidance on using AI tools for faster code dev, implementing data governance & more. Read here
-
Training Models at Scale: 10K+ H100 Infrastructure Guide
By
–
quick mini-post I wrote for @francoisfleuret broadly summarizing the things one needs to do to train models on 10k+ H100s.
-
Scaling AI Training Across Thousands of H100 GPUs
By
–
There's three parts. 1. Fitting as large of a network and as large of a batch-size as possible onto the 10k/100k/1m H100s — parallelizing and using memory-saving tricks.
2. Communicating state between these GPUs as quickly as possible
3. Recovering from failures (hardware, -
Tokenization Fundamentals: Mastering Text Processing for Generative AI
By
–
Tokenization — turning text into a sequence of integers — is a key part of generative AI, and most API providers charge per million tokens. How does tokenization work? Learn the details of tokenization and RAG optimization in Retrieval Optimization: From Tokenization to Vector… pic.twitter.com/AsOI8UrXkn
— Andrew Ng (@AndrewYNg) 2 octobre 2024Tokenization — turning text into a sequence of integers — is a key part of generative AI, and most API providers charge per million tokens. How does tokenization work? Learn the details of tokenization and RAG optimization in Retrieval Optimization: From Tokenization to Vector
-
Face Detection Study Reduces False Positives for Autonomous Systems
By
–
The study could potentially help face detection systems by reducing false positives, which might impact self-driving cars, human-computer interaction, & robotics. The dataset & models could also help areas like product design.