It's terrible for most regular people. We are memory poor — MoE models make this even worse. It may be cute, but it's totally impractical. Dense models are far more usable in practice.
MoE Models Memory Inefficiency Dense Models Practicality
By
–
By
–
It's terrible for most regular people. We are memory poor — MoE models make this even worse. It may be cute, but it's totally impractical. Dense models are far more usable in practice.