Smart technical teams are already starting to confirm that the techniques and resulting cost savings are real.
MACHINE LEARNING
-
GAN Test of Time Talk Recording Now Available
By
–
The recording of the GAN test of time talk by @dwf is now publicly available: https://
neurips.cc/virtual/2024/t
est-of-time/105032
… -
How DeepSeek R1 Reasoning Model Works
By
–
This is how DeepSeek r1 reasoning model works behind the scenes. A 3-turn chain of thought reflecting on 3 questions.
-
Nature publishes evolutionary model merging optimization research
By
–
論文「Evolutionary Optimization of Model Merging Recipes」が論文誌「Nature Machine Intelligence」に採択され本日掲載されました。最新バージョンでは本アプローチをさらに実証する新たな実験結果を含んでいます。ぜひ以下からご覧ください。 https://
nature.com/articles/s4225
6-024-00975-8
… Sakana -
Nature Publishes Evolutionary Optimization Model Merging Paper
By
–
We’re pleased to announce that our paper, “Evolutionary Optimization of Model Merging Recipes,” has been accepted to Nature Machine Intelligence and published today! The latest version includes new experimental results that further validate our approach. https://
nature.com/articles/s4225
6-024-00975-8
… -
GRPO motivation: computational and memory efficiency clarification
By
–
Awesome, thanks a lot for this direct head-to-head comparison! Actually, after reading the R1 paper, wasn't the pure motivation behind GRPO computational (/memory) efficiency?
-
Adding Official OpenAI Transformer Weights to Notebook
By
–
Ah nice! I thought they only had the transformer-library-formatted weights on the hub! In that case, I can actually add that to the notebook directly. Cool stuff! (Btw the reason why I opted for the OpenAI Tf ones was that they were the original/official ones)
-
Deep Learning’s Legendary Appetite for Compute Resources
By
–
I don't have too too much to add on top of this earlier post on V3 and I think it applies to R1 too (which is the more recent, thinking equivalent). I will say that Deep Learning has a legendary ravenous appetite for compute, like no other algorithm that has ever been developed
-
DeepSeek-R1 Training Cost: Missing R1 Distillation Expenses
By
–
The real question: What is the DeepSeek-R1 training cost? The $5.567M DeepSeek cost is missing the cost of training the R1 model to get distilled data. Table 9 in the DeepSeek v3 paper shows that the R1 distillation step is critical for quality. The R1 paper doesn't talk about
-
Pre-training remains superior to expensive knowledge injection methods
By
–
Interesting! But knowledge is still best instilled via pre-training. "Relatively" cheap still remains very expensive and cumbersome, even for 7B models, compared to just adding some more embeddings to a database.