10/ MoE-Mamba – an approach to efficiently scale LLMs by combining state space models (SSMs) with Mixture of Experts (MoE); MoE-Mamba, outperforms both Mamba and Transformer-MoE.
MoE-Mamba: Scaling LLMs with State Space Models and Mixture of Experts
By
–
