Always go back to basics: LatentMoE was probably inspired by MLA, which was inspired by LoRA, which was inspired by SVD, which was inspired by eigenvalue decomposition.
Lineage of techniques: from eigen decomposition to LatentMoE
By
–
By
–
Always go back to basics: LatentMoE was probably inspired by MLA, which was inspired by LoRA, which was inspired by SVD, which was inspired by eigenvalue decomposition.