Thought the same thing and was trying to reimplement ReMoE over the past days… Achieving the same level of utilization as the default codebase, and thus beating the wall-clock record, will not be trivial.

→ View original post on X — @alexjc