You mean alternatives to self-attention specifically (versus parameter efficient finetuning etc.)? I think none of them stood the test of time. I think the problem is that they are all approximations. The relatively recent FlashAttention mechanism is super popular though.
@rasbt
-
Reward Model Training vs User Feedback: Preferences and Finetuning
By
–
Good question. In their original finetuning, they train a reward model based on relative preference (rankings among multiple responses). And from the user feedback, there's only thumbs up & down. You can probably use that for supervised finetuning I guess.
-
Reasons Behind OpenAI Models’ Superior Performance Remain Unknown
By
–
Yeah. And we still don't know why OpenAI models are better: 1. More and better data?
2. Larger pretrained base model?
3. RLHF instead of supervised finetuning? My guess is it's probably all of these above. -
Dataset Importance for LLM Fine-tuning: Camels Paper Insights
By
–
And the recent "How Far Can Camels Go" paper (
https://
arxiv.org/abs/2306.04751) is a nice example of how important the dataset for LLM finetuning is. -
SLM Acronym Ambiguity: Small Language Model or Subspace Learning Machine
By
–
SLM as in Small Language Model or Subspace Learning Machine?
-
Training Models to Convergence Remains Essential
By
–
The day where I/we can’t train a model until convergence anymore will be a sad day.
-
Model Distillation Performance Trade-offs in Large Language Models
By
–
But yeah, modeling performance-wise you'll probably take a hit. Same thing that happened when they distilled ChatGPT GPT-3.5 and later GPT-4
-
Making LLM serving affordable through pruning and quantization
By
–
Fair point, but I think the appeal is more in trying to make running LLM more affordable (via pruning and quantization) to reduce the number of GPUs required for serving.
-
Standard Practice for Pruning Linear Layers in Neural Networks
By
–
I haven't seen that yet. I think the standard is to prune all linear layers, this includes the QVK of the attention layers and the FC layers (except the embedding and output layers of course).
-
Pruning Large Language Models Without Retraining Using Activation Norms
By
–
LLMs are usually too large for most contexts, but creating pruned versions of a model usually require retraining. Here's a new straightforward alternative based on computing element-wise product between the weight magnitude and norm of input activations: https://
arxiv.org/abs/2306.11695