i suspect this is a contingent but not necessary truth. if we refocused routing objective to be topic experts, or sparse-upscaled topic experts, then we can also achieve plug and play experts or frankenmerged MoEs
Routing Objectives and Modular Expert Systems in MoEs
By
–