Excited to announce that we’ve raised $1.3B to build one of the largest clusters in the world and turbocharge the creation of Pi, your personal AI. https://
forbes.com/sites/alexkonr
ad/2023/06/28/inflection-ai-raises-1-billion-for-chatbot-pi/
…
AI
-
Inflection AI raises $1.3B for massive AI cluster and Pi
By
–
-
Low-Rank Matrices in Task-Specific Training vs Pretraining
By
–
I think we are talking about the same thing. That would only work after training. It's based on the assumption that the matrices have low rank for a specific target task but are not low rank in general (for the pretraining tasks).
-
LoRA Weight Matrix Factorization During Fine-tuning Explained
By
–
You mean as in LoRA (low-rank adaptation)? You are factorizing the weight matrices but only during finetuning. The self-attention mechanism computation is still the same (if you ignore that the weight matrices are different).
-
GPU Heat and Noise Issues Require Laundry Room Setup
By
–
I considered it still too hot and noisy to be in the same room with. I probably need to put a rack in the laundry room if I still want ML GPUs now.
-
Nature Editorial: Constructive Discussion on AI Existential Risks
By
–
This week's @Nature editorial "Fearmongering narratives about existential risks are not constructive. Serious discussion about actual risks and action to contain them, are." https://
nature.com/articles/d4158
6-023-02094-7
… -
Multi-query attention experiences revival with Falcon model
By
–
Also, multi-query attention has a little comeback (thanks to Falcon)
-
Self-Attention Alternatives Failed; FlashAttention Gains Popularity
By
–
You mean alternatives to self-attention specifically (versus parameter efficient finetuning etc.)? I think none of them stood the test of time. I think the problem is that they are all approximations. The relatively recent FlashAttention mechanism is super popular though.
-
AI Safety Needs More Resources Amid Rapid Development
By
–
“Right now there are 99 very smart people trying to make #AI better and one very smart person trying to figure out how to stop it taking over and maybe you want to be more balanced.”—
@geoffreyhinton https://
news.com.au/technology/inn
ovation/inventions/smarter-than-us-ai-godfathers-grim-warning-for-the-future/news-story/58684beaaa114b09d2a430dd08556818
… -
Reward Model Training vs User Feedback: Preferences and Finetuning
By
–
Good question. In their original finetuning, they train a reward model based on relative preference (rankings among multiple responses). And from the user feedback, there's only thumbs up & down. You can probably use that for supervised finetuning I guess.
-
Reasons Behind OpenAI Models’ Superior Performance Remain Unknown
By
–
Yeah. And we still don't know why OpenAI models are better: 1. More and better data?
2. Larger pretrained base model?
3. RLHF instead of supervised finetuning? My guess is it's probably all of these above.