I think so. In their 3.35B Tiny Aya report, they say "We use parallel Transformer blocks, which lead to a signifi-
cant improvement in training efficiency without hurting model quality."

→ View original post on X — @rasbt