My understanding is that @percyliang
's prefix tuning approach prepended activations to *every* layer, but yours only prepends to the first layer — is that right?
@jeremyphoward
-
Prefix Tuning Approaches: Layer-wise Activation Prepending Comparison
By
–
-
Prefix Tuning Initialization with Real Tokens and DoRA
By
–
Is the trick of initializing with real tokens the secret to making all prefix tuning approaches competitive? BTW @EyubogluSabri I don't see DoRA mentioned in your paper – I wonder if that would close the gap a bit?
-
LoRA vs DoRA vs Prefix Tuning: Fine-tuning Method Effectiveness
By
–
The big open question for me is why LoRA (and particularly DoRA) has been the most successful for fine-tuning, but it worked so badly for this use case. Or maybe the opposite – why did prefix tuning work so well?
-
Meta’s AI Talent Loss: Strategic Consequences of Leadership Decisions
By
–
If Zuck hadn't laid off Erik's team of exceptional AI talent a few years ago, they would have less of an AI talent problem today…
-
Faster Academic Citation Lookup Using PDF Upload
By
–
BTW this is way faster than manually going to the footnotes of a paper to see what each number refers to — even when it turns out it cites a paper you're familiar with; just copy/paste the sentence including cite numbers into a thread which you've uploaded the PDF to.
-
Claude Limited by Context Window in Research Tasks
By
–
Claude not able to continue my research chat about context compression papers because it ran out of context because it doesn't use context compression.
-
OpenAI Claims Model Parity: Awaiting Updated Aider Evaluations
By
–
Note that @OpenAI claims that this model is identical to the more expensive one, but we should wait for updated aider evals to be sure
-
O3 Emerges as Strong Coding Option with Major Price Cut
By
–
Wow O3 is a *very* strong option for coding now. I've updated @paulgauthier
's latest Aider eval with this O3 80% price cut – check out O3 in 3rd place, but cheaper and faster than Gemini Pro now:
