Wow, a new blogpost from the GOAT Sebastian Raschka! Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention
RESEARCH
-
Google researchers: can’t imagine publishing transformer paper now, return to academia?
By
–
Google researchers say they can’t imagine being allowed to publish the transformer paper now. Time to return to academia?
-
Yoshua Bengio’s Large-Scale Graduate Student Descent Breakthrough
By
–
The key AI breakthrough of the last 20 years, due to Yoshua Bengio, was large-scale graduate student descent.
-
RSI and continual learning as barriers to AI takeoff
By
–
So the two most obvious barriers to some sort of true AI takeoff are robust RSI (AI acting as an independent AI researcher, rather than “merely” a multiplier of human effort) and continual learning. Either would represent a major change in trajectory for AI development.
-
Experiments Used GPT-4 and GPT-4o; Newer Models Could Improve Results
By
–
And, yes, our experiments used a mix of GPT-4 & GPT-4o (publishing takes awhile). I think we would see much larger results with more recent models, let alone recent agentic tools.
-
LLM traces masquent un planificateur myope
By
–
“Extracting Search Trees from LLM Reasoning Traces Reveals Myopic Planning” Reasoning models can write traces that look like real tree search, but this paper shows their decisions are mostly driven by shallow one-step evaluation. They extract search trees from LLM CoT in
-
Oxford AI4Science research on LLM long-horizon reasoning
By
–
Long-horizon reasoning is one of the largest obstacles in LLMs. In our latest AI4Science talk, Sumeet (
@sumeetrm
) and Charlie (
@CharlieLondon02
) from Oxford discussed H1 and LongCoT, two projects focused on measuring and improving how models reason over long chains of steps. -
SlimQwen: Technical Research on Pruning and Distillation for MoE Models
By
–
“SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training” This new Qwen paper shows that pruning a pretrained MoE is much better than training the smaller MoE from scratch. All you need to do is prune depth, width, and experts, preserve some experts
-
Avoid training AI on its own actions to prevent delusions
By
–
Actions persist in history and can affect the world. However, we shouldn’t train on our own actions. Training on our own actions as if they were evidence is what can cause delusions.
-
On model choice and training dynamics
By
–
You don’t remove any information. You just use the right model. The action distribution is a delta function for your own actions. You still train on how the world is affected by or reacts to your actions