AI Dynamics

"Path-Constrained Mixture-of-Experts" MoE models may be wasting signal by routing too independently. In a standard MoE, each layer picks experts independently, so across L layers with N experts you get N^L possible expert paths. That path space is so huge that most routes barely get any learning signal. So this paper PathMoE fixes this with a very simple idea: share router parameters across small blocks of consecutive layers, so tokens follow more coherent paths through the network instead of constantly changing paths. Not only are the paths now interpretable, it opens up new ideas like global path design. On a 0.9B MoE, it improves average downstream accuracy by +2.1 points, and around 4% improvements on a 16B model. Routing is cleaner too, 79% vs 48% routing consistency across layers, 11% lower routing entropy, and 22.5x more robustness to routing perturbations, all without needing an auxiliary load-balancing loss!

→ View original post on X — @askalphaxiv, 2026-04-01 17:53 UTC