The big open question for me is why LoRA (and particularly DoRA) has been the most successful for fine-tuning, but it worked so badly for this use case. Or maybe the opposite – why did prefix tuning work so well?

→ View original post on X — @jeremyphoward