We experimented with alternatives to our final Jamba architecture, including Mamba-2 (which was released a few months after the original Jamba). However, we found that in a hybrid architecture, the Mamba-1-Attention combination outperforms both Mamba-2-Attention and pure Mamba-2.

→ View original post on X — @ai21labs