RNNs are not dead yet In fact, they are coming back with a vengeance recently. Very nice paper about “The Illusion of State in State-Space Models” (
https://
arxiv.org/abs/2404.08819) and thread
@hardmaru
-
RNNs Making a Comeback: State-Space Models Analysis
By
–
-
Model Fine-Tuning for Human Preferences
By
–
Your model, after some fine-tuning for human preferences.pic.twitter.com/lzq5WAqP1b
— hardmaru (@hardmaru) 15 avril 2024Your model, after some fine-tuning for human preferences.
-
OpenAI Expansion to Tokyo: AI Hub Growth in Japan
By
–
Great to see more companies like OpenAI also expanding to Japan. Tokyo is going to become a city with a significant presence in AI. Tokyo has always attracted the ‘right’ kind of people: creative, curious, weird and eccentric. Official announcement here:
-
NeurIPS Introduces Preschool Track for Machine Learning Research
By
–
Announcing NeurIPS Preschool Track This year, we invite preschoolers to submit machine learning research papers.
-
Collective Intelligence Across Biological Scales and Swarm Behavior
By
–
“Collective intelligence is not only the province of groups of animals, and that an important symmetry exists between the behavioral science of swarms and the competencies of cells and other biological systems at different scales.”
-
H100 GPU Pricing in Japan at $2.6 per Hour
By
–
H100 at ~$2.6/hour in Japan. I can get cheaper options, but we’re talking about only a few thousand bucks of savings at this scale.
-
Avoid Multi-Year Compute Contracts, Focus on Innovation Instead
By
–
As an AI R&D company, the last thing I’d do is to lock myself up in multi-year expensive compute contracts. Those crazy fixed-costs will ultimately destroy many so-called “GenAI” startups. Focus on innovation instead. My comment on @steph_palazzolo piece: https://
theinformation.com/articles/why-t
wo-models-are-better-than-one?commentId=22938
… -
Sakana AI’s cost-efficient model merging with limited GPU resources
By
–
“At the time, @SakanaAILabs only had 16 GPUs, which cost $30K per month. That’s pennies compared to the $100M+ it took to train advanced models like GPT-4… Model merging isn’t perfect though. @SakanaAILabs is one of only a few firms who have attempted to automate this process.”
-
Reusing Old NeurIPS Paper Template After Three Years
By
–
The last time I published a NeurIPS paper was back in 2021, so I just reused my old template from 3 years ago. I’m not sure we will submit to NeurIPS, since I heard that the process it is quite difficult these days 🙂