Calling all Mathletes, this one is for you. We’ve been asked to show the math behind our MoE claims. So we did. Our analysis confirms: On GPUs, expert parallelism creates severe communication overheads that dwarf computation and make MoE training painfully slow. At

→ View original post on X — @cerebras