I haven't checked but that sounds good indeed, you might be able to directly do a phi-3 MoE with this

→ View original post on X — @maximelabonne