quick mini-post I wrote for @francoisfleuret broadly summarizing the things one needs to do to train models on 10k+ H100s.
COMPUTING
-
Scaling AI Training Across Thousands of H100 GPUs
By
–
There's three parts. 1. Fitting as large of a network and as large of a batch-size as possible onto the 10k/100k/1m H100s — parallelizing and using memory-saving tricks.
2. Communicating state between these GPUs as quickly as possible
3. Recovering from failures (hardware, -
MongoDB Schema Modeling: Beyond Schemaless Myth
By
–
Challenge accepted! I think my point would be that you can model your data in MongoDB as you would in PG. there are good reasons why documents could get you farther, depending on the workload, but this idea of "schemaless" MongoDB has always been wrong. There's always schema.
-
IBM Establishes First Quantum Data Center in Europe
By
–
[#Article] IBM Expands Quantum Footprint with First European Datacenter https://actuia.com/actualite/ibm-etend-son-empreinte-quantique-avec-un-premier-datacenter-europeen/
… #AI #artificialintelligence -
Everyday Transformers Optimizations: KV Cache, FlashAttention, PagedAttention
By
–
This blog post is really cool: to understand everyday Transformers optimizations like KV cache, FlashAttention or PagedAttention: https://
astralord.github.io/posts/transfor
mer-inference-optimization-toolset/
… Image below is the interactive visualization for KB cache! -
AlphaChip: How AI Redefines Chip Design
By
–
[#Article] AlphaChip: How DeepMind's AI Redefines Chip Design Standards https://actuia.com/actualite/alphachip-comment-lia-de-deepmind-redefinit-les-normes-de-la-conception-des-puces/
… #AI #ArtificialIntelligence -
Google Demos LookupViT: Sub-Quadratic Attention at ECCV2024
By
–
Want a scalable and robust foundational model that also saves computational costs? Drop by the #ECCV2024 Google Booth at 10:30am CEST where Googlers Rajat Koner, Sujoy Paul and Gagan Jain will demo LookupViT, an information compression idea leading to sub-quadratic attention.
-
Amazon’s 750,000 AI Robots Transform E-commerce Operations
By
–
The Amazing Ways Amazon Is Using AI Robots Amazon is revolutionizing #ecommerce with its fleet of over 750,000 #AI-powered #robots. From mobile helpers to advanced sorting systems, these innovations are transforming #productivity, #safety, and customer service across #Amazon's
-
AI Productivity Gains: From Consultants to Quantum Computing
By
–
Les sources du FT sont ici : https://
computerworld.com/article/354342
1/ai-could-be-a-good-replacement-for-bad-bosses-and-expensive-consultants.html
… https://
next.ink/brief_article/
ibm-ouvre-son-premier-datacenter-quantique-en-europe/
… https://
bloomberg.com/news/articles/
2024-10-01/fed-s-cook-sees-ai-boosting-productivity-how-much-still-unclear
… https://
techcrunch.com/2024/10/01/mic
rosoft-copilot-can-now-read-your-screen-think-deeper-and-speak-aloud-to-you/
… https://
zdnet.fr/actualites/la-
startupeuse-julie-huguet-nouveau-visage-de-la-french-tech-398673.htm#xtor=RSS-1
… -
Disruptive Technology Set to Transform Lunar Mission Plans
By
–
Exciting. Watch this space! I can already sense this will be disruptive to plans to get to and from (and supply) the moon.