For more context, since I didn't see a link in the article, are we talking about the "The Curse of Recursion: Training on Generated Data Makes Models Forget" paper? (
https://
arxiv.org/abs/2305.17493) And the claim is that the findings in this paper don't / won't hold in the future?
@rasbt
-
Questioning the Curse of Recursion Paper’s Future Relevance
By
–
-
OpenAssistant’s shift from RLHF to supervised instruction-finetuning
By
–
So on that note, OpenAssistant's earlier models used RLHF for instruction-finetuning (
https://
huggingface.co/OpenAssistant/
oasst-rlhf-2-llama-30b-7k-steps-xor
…); The later one seem to use supervised instruction-finetuning. Maybe @ykilcher has some insights whether RLHF wasn't worth the effort vs supervised? -
RLHF versus Supervised Instruction Tuning Clarification
By
–
Isn’t RLHF also instruction finetuning? You mean supervised instruction tuned?
-
Data Compensation for Commercial AI Training: Need Government Guidance
By
–
I am in strong favor of compensating those whose data was used for training if it’s purely for commercial (vs research or educational) purposes. But wouldn’t it make sense to have a state-/government-issued guidance (like Japan did) before wasting time & money on these lawsuits?
-
FlashAttention and KV Caching for Improved Model Performance
By
–
No, absolutely. But I think for this it's maybe worthwhile adopting methods that don't negatively impact modeling performance, like FlashAttention and kv caching.
-
State-of-the-Art Models First, Efficiency Through Distillation Later
By
–
For most, the goal is to push the state-of-the-art and show that these models can achieve impressive results. You can always distill models later to make them more efficient.
-
Trade-offs Between Efficiency and Model Performance in Pretraining
By
–
You are right. But yeah the point is that these are based on assumptions and produce approximations. You'd hurt modeling performance and people don't want that, which is why the "efficient alternatives that do approximations" for/during pretraining haven't caught on.
-
Low-Rank Matrices in Task-Specific Training vs Pretraining
By
–
I think we are talking about the same thing. That would only work after training. It's based on the assumption that the matrices have low rank for a specific target task but are not low rank in general (for the pretraining tasks).
-
LoRA Weight Matrix Factorization During Fine-tuning Explained
By
–
You mean as in LoRA (low-rank adaptation)? You are factorizing the weight matrices but only during finetuning. The self-attention mechanism computation is still the same (if you ignore that the weight matrices are different).
-
Multi-query attention experiences revival with Falcon model
By
–
Also, multi-query attention has a little comeback (thanks to Falcon)