Yes! It would be interesting whether this happens in GQA:SWA models only though or also in MLA or Gated DeltaNet models, for example.
Comparative analysis of GQA, MLA, and Gated DeltaNet model architectures
By
–
By
–
Yes! It would be interesting whether this happens in GQA:SWA models only though or also in MLA or Gated DeltaNet models, for example.