It can be unintuitive why the Transformer-style MoE (in Mixtral/GPT4) has inference benefits.
Dima simplifies it with a clear explanation showcasing that MoE help inference once there's sufficient volume of requests (which hopefully are diverse enough that they don't hit the same

→ View original post on X — @soumithchintala,