Agreeing and extending @sashamtl
, there are lots of reasons why OpenAI likely isn’t open about training data: copyright, bias, and also questions about generalization & data contamination. I too salute @abebab
’s great detective work, including an important new paper coming soon.
RESEARCH
-
OpenAI Training Data Transparency: Copyright, Bias, and Generalization Issues
By
–
-
FlashAttention and KV Caching for Improved Model Performance
By
–
No, absolutely. But I think for this it's maybe worthwhile adopting methods that don't negatively impact modeling performance, like FlashAttention and kv caching.
-
State-of-the-Art Models First, Efficiency Through Distillation Later
By
–
For most, the goal is to push the state-of-the-art and show that these models can achieve impressive results. You can always distill models later to make them more efficient.
-
Trade-offs Between Efficiency and Model Performance in Pretraining
By
–
You are right. But yeah the point is that these are based on assumptions and produce approximations. You'd hurt modeling performance and people don't want that, which is why the "efficient alternatives that do approximations" for/during pretraining haven't caught on.
-
Low-Rank Matrices in Task-Specific Training vs Pretraining
By
–
I think we are talking about the same thing. That would only work after training. It's based on the assumption that the matrices have low rank for a specific target task but are not low rank in general (for the pretraining tasks).
-
LoRA Weight Matrix Factorization During Fine-tuning Explained
By
–
You mean as in LoRA (low-rank adaptation)? You are factorizing the weight matrices but only during finetuning. The self-attention mechanism computation is still the same (if you ignore that the weight matrices are different).
-
Nature Editorial: Constructive Discussion on AI Existential Risks
By
–
This week's @Nature editorial "Fearmongering narratives about existential risks are not constructive. Serious discussion about actual risks and action to contain them, are." https://
nature.com/articles/d4158
6-023-02094-7
… -
Multi-query attention experiences revival with Falcon model
By
–
Also, multi-query attention has a little comeback (thanks to Falcon)
-
Self-Attention Alternatives Failed; FlashAttention Gains Popularity
By
–
You mean alternatives to self-attention specifically (versus parameter efficient finetuning etc.)? I think none of them stood the test of time. I think the problem is that they are all approximations. The relatively recent FlashAttention mechanism is super popular though.
-
AI Safety Needs More Resources Amid Rapid Development
By
–
“Right now there are 99 very smart people trying to make #AI better and one very smart person trying to figure out how to stop it taking over and maybe you want to be more balanced.”—
@geoffreyhinton https://
news.com.au/technology/inn
ovation/inventions/smarter-than-us-ai-godfathers-grim-warning-for-the-future/news-story/58684beaaa114b09d2a430dd08556818
…