DiffusionBlocks: Training Neural Networks One Block at a Time
@hardmaru
-
Novel Block-Based Backprop Reduces AI Training Memory
By
–
For over a decade, we’ve accepted that end-to-end backprop is the only way to train deep networks. But holding the entire network in memory all at once is why AI training is hitting a resource wall.
— hardmaru (@hardmaru) 27 mai 2026
We found a new way to break the network into blocks and train them… https://t.co/vxyMR6goTDFor over a decade, we’ve accepted that end-to-end backprop is the only way to train deep networks. But holding the entire network in memory all at once is why AI training is hitting a resource wall. We found a new way to break the network into blocks and train them
-
AI Forecasting Scientific Progress: Capabilities and Limitations
By
–
Forecasting Scientific Progress with Artificial Intelligence https://
arxiv.org/abs/2605.22681 Turns out AI is just as bad at forecasting biology and physics breakthroughs as we are. To be fair, most breakthroughs cannot be predicted. Science is more like an evolutionary search process. -
AI Tools Make Engineers More Productive, Expand SWE Teams
By
–
People keep asking if AI will replace software engineers. I believe the exact opposite. Thanks to the Jevons paradox, AI tools are making great engineers 10x more productive, allowing us to tackle much harder, larger-scale problems. We’re expanding our SWE teams at @SakanaAILabs
-
Research on Sparser, Faster, and Lighter Transformer Language Models
By
–
Sparser, Faster, Lighter Transformer Language Models
-
Historical Research on LSTMs for Music Improvisation
By
–
The agent also reimplemented the “Blues Improvisation” experiment by @douglas_eck and @SchmidhuberAI in 2002 which show that LSTMs can learn temporal structure in music.
— hardmaru (@hardmaru) 12 mai 2026
Finding temporal structure in music: Blues improvisation with LSTM recurrent networkshttps://t.co/2oU4iIcJEr pic.twitter.com/o4Eq4KqSGqThe agent also reimplemented the “Blues Improvisation” experiment by @douglas_eck and @SchmidhuberAI in 2002 which show that LSTMs can learn temporal structure in music. Finding temporal structure in music: Blues improvisation with LSTM recurrent networks https://
sferics.idsia.ch/pub/juergen/20
02_ieee.pdf
… -
Reproducing Schmidhuber’s Research Papers Using AI Coding Assistants
By
–
Reproducing all of Schmidhuber’s papers (1990-2025) using an AI coding assistant.
— hardmaru (@hardmaru) 9 mai 2026
Cool project by @yaroslavvb! It even reproduced the “World Models” paper by me and @SchmidhuberAI with a toy env, with a full VAE + RNN world model implementation.
Project: https://t.co/sgQG5umNEm pic.twitter.com/iKMFN7ti9zReproducing all of Schmidhuber’s papers (1990-2025) using an AI coding assistant. Cool project by @yaroslavvb
! It even reproduced the “World Models” paper by me and @SchmidhuberAI with a toy env, with a full VAE + RNN world model implementation. Project: https://
github.com/cybertronai/sc
hmidhuber-problems/blob/main/VISUAL_TOUR.md
… -
Custom CUDA Kernels for TwELL Format on H100 GPUs Released
By
–
If you want to look under the hood at the actual custom CUDA kernels and see exactly how we implemented the TwELL format for H100 GPUs, we’ve released the reference code. GitHub: https://
github.com/SakanaAI/spars
er-faster-llms
…
Blog: https://
pub.sakana.ai/sparser-faster
-llms/
… -
Hardware limitations on sparse neural activation in LLMs
By
–
The human brain is incredibly efficient because it only activates the specific neurons needed for a thought. Modern LLMs naturally try to do this too (> 95% of neurons in feedforward layers stay silent for any given word), but our hardware punishes them for it. One of the most
-
Sakana Fugu: Multi-Agent Orchestration System as Foundation Model
By
–
Sakana Fugu: A Multi-Agent Orchestration System as a Foundation Model