all experts are merged into an MoE. The anchor model ensures that all experts learn to coordinate, even though they are never trained on the joint dataset. this is rocket science wtf

→ View original post on X — @swyx,