When will the Transformer be superseded?
MACHINE LEARNING
-
Bytedance GRPO Paper Challenges Optimization Approach Efficiency
By
–
Following an intense weekend of GRPO discussion, Bytedance put out a paper on why GRPO is NOT optimal! Similar to the classic knapsack problem, they suggest that each exploration has a “value” and “cost”, which needs to be adaptively distributed. This yields 20-40% more signal
-
Compile Effects on Neural Network Layers and MoE Architecture
By
–
Interesting, interesting. I thought compile would rather affect the nn.Linear layers vs nn.Embedding as the latter is a lookup and not doing any actual math/compute. Curious now and have to run some experiments later. But since you mentioned MoE, also I am confused now where you
-
Hugging Face Transformers: Managing 1M Lines Python Code
By
–
With 1,000,000+ lines of Python and 400+ models How do we keep @huggingface Transformers sane? > Our design tenets: one-model-one-file, modular shards, minimal API, no breakage Peek under the hood on one of the biggest and most-used open ML codebases out there
-
AI Systems Solving Voice Recognition and Cocktail Party Problem
By
–
maybe that’s the use case for current/next gen AI systems. I mean voice inputs have been pretty much solved years ago. But there are still false positives, and newer systems may solve the cocktail party problem here.
-
torch.compile Performance with nn.Embedding Layers
By
–
Interesting, do you mean torch.compiling here? And if yes, why/how does that work in terms of speed-up since nn.Embedding is basically just a tensor for cheap look-up? (But also, if you are already compiling the model, why would you skip the embedding layer in the first place?)
-
Introducing Continuous Thought Machines New Framework
By
–
Introducing Continuous Thought Machineshttps://t.co/Fjwl82PZ9E
— hardmaru (@hardmaru) 6 octobre 2025
(Blog post from earlier this year.)
Summary tweet:https://t.co/bOGIDVDZodIntroducing Continuous Thought Machines https://
sakana.ai/ctm/
(Blog post from earlier this year.) Summary tweet: -
Internal metrics and taxonomy in AI model evaluation
By
–
You are not wrong but that was a deliberate as I covered the internal metrics (loss, perplexity, rewards) many times before . Thanks for sharing the taxonomy paper btw!
-
Transformers Shape Language Patterns Like Marbles
By
–
We say “Transformers predict the next word,” but really what’s happening is: They’re reshaping a whole field of relationships until the pattern feels stable – like a marble rolling down until it rests in the lowest valley. That “valley” is the lowest-entropy shape of language –
-
Continuous Thought Machines Spotlight at NeurIPS 2025
By
–
Proud to share that “Continuous Thought Machines” will be presented as a spotlight at #NeurIPS2025 ✨
— hardmaru (@hardmaru) 6 octobre 2025
Updated paper: https://t.co/ZWiidb1eTX
Interactive Demo: https://t.co/d6hqid60t7
GitHub: https://t.co/O0FS9jM8gk https://t.co/Zgv6TIFybSProud to share that “Continuous Thought Machines” will be presented as a spotlight at #NeurIPS2025 Updated paper: https://
arxiv.org/abs/2505.05522
Interactive Demo: https://
pub.sakana.ai/ctm/
GitHub: https://
github.com/SakanaAI/Conti
nuous-Thought-Machines
…