"Slicing and Dicing: Configuring Optimal MoE" MoE models have a huge design space, but most choices are overcomplicated. This paper runs 2,000+ pretraining runs up to 6.6B params and finds the key knobs are mostly expert count and expert granularity. Adding more inactive
Technical Analysis of Optimal Mixture-of-Experts Model Configuration
By
–
