That’s absolutely true and a good point. I remember multiquery attention struggles when trying to port LoRA from LLaMA to Falcon.
Multiquery Attention Challenges with LoRA Porting Between Models
By
–
By
–
That’s absolutely true and a good point. I remember multiquery attention struggles when trying to port LoRA from LLaMA to Falcon.