i suspect this is a contingent but not necessary truth. if we refocused routing objective to be topic experts, or sparse-upscaled topic experts, then we can also achieve plug and play experts or frankenmerged MoEs

→ View original post on X — @swyx