all experts are merged into an MoE. The anchor model ensures that all experts learn to coordinate, even though they are never trained on the joint dataset. this is rocket science wtf
Merged Experts in MoE: Coordination Without Joint Training
By
–
By
–
all experts are merged into an MoE. The anchor model ensures that all experts learn to coordinate, even though they are never trained on the joint dataset. this is rocket science wtf