What's coming next: → Adaptive expert count (dynamically add/remove experts during training)
→ Cross-model expert sharing (reuse specialists across different models)
→ Hierarchical MoE (experts that route to sub-experts)
→ Expert distillation (compress MoE knowledge back

→ View original post on X — @godofprompt