It's a dense model. Apparently, they merged weights from different versions of Gemma 2.

→ View original post on X — @maximelabonne