You are right. But yeah the point is that these are based on assumptions and produce approximations. You'd hurt modeling performance and people don't want that, which is why the "efficient alternatives that do approximations" for/during pretraining haven't caught on.
@rasbt
-
Low-Rank Matrices in Task-Specific Training vs Pretraining
By
–
I think we are talking about the same thing. That would only work after training. It's based on the assumption that the matrices have low rank for a specific target task but are not low rank in general (for the pretraining tasks).
-
LoRA Weight Matrix Factorization During Fine-tuning Explained
By
–
You mean as in LoRA (low-rank adaptation)? You are factorizing the weight matrices but only during finetuning. The self-attention mechanism computation is still the same (if you ignore that the weight matrices are different).
-
Multi-query attention experiences revival with Falcon model
By
–
Also, multi-query attention has a little comeback (thanks to Falcon)
-
Self-Attention Alternatives Failed; FlashAttention Gains Popularity
By
–
You mean alternatives to self-attention specifically (versus parameter efficient finetuning etc.)? I think none of them stood the test of time. I think the problem is that they are all approximations. The relatively recent FlashAttention mechanism is super popular though.
-
Reward Model Training vs User Feedback: Preferences and Finetuning
By
–
Good question. In their original finetuning, they train a reward model based on relative preference (rankings among multiple responses). And from the user feedback, there's only thumbs up & down. You can probably use that for supervised finetuning I guess.
-
Reasons Behind OpenAI Models’ Superior Performance Remain Unknown
By
–
Yeah. And we still don't know why OpenAI models are better: 1. More and better data?
2. Larger pretrained base model?
3. RLHF instead of supervised finetuning? My guess is it's probably all of these above. -
Dataset Importance for LLM Fine-tuning: Camels Paper Insights
By
–
And the recent "How Far Can Camels Go" paper (
https://
arxiv.org/abs/2306.04751) is a nice example of how important the dataset for LLM finetuning is. -
SLM Acronym Ambiguity: Small Language Model or Subspace Learning Machine
By
–
SLM as in Small Language Model or Subspace Learning Machine?
-
Training Models to Convergence Remains Essential
By
–
The day where I/we can’t train a model until convergence anymore will be a sad day.