Hard to say, but I think it may be independent of MHA vs GQA. Also, Olmo 3 7B uses MHA, and 32B uses GQA.

→ View original post on X — @rasbt