Is the trick of initializing with real tokens the secret to making all prefix tuning approaches competitive? BTW @EyubogluSabri I don't see DoRA mentioned in your paper – I wonder if that would close the gap a bit?

→ View original post on X — @jeremyphoward