@avshalomm solved it by utilizing the fact that actually, there is no interaction between different tokens in the MoE block, so we can iterate over the long context in chunks. This was also merged and now fixed in vLLM

→ View original post on X — @ai21labs,