One bug in bitsandbytes caused QLoRA to consume an extra *read notes* 4888 GB of VRAM. Even with QLoRA, fine-tuning Llama 3 405B on long sequence lengths ain't cheap.
RESEARCH
-
Conference Evolution: AI Impact Since GPT-3 Era
By
–
I’ve been attending these conferences since 1997 so I have a sense for their history, and I disagree that it was like this pre-Covid or pre-GPT3. Also, I’m not saying that this is a good or bad thing (no judgement), it just is. I find it interesting, and worth reflecting on.
-
Decaf: First Open Source AlexNet for Feature Reuse
By
–
Thanks so much for this award (makes me feel old )! I am honored to be sharing this award with amazing coauthors Decaf was the first open source version of AlexNet, and tested whether features learned by this amazing ImageNet classifier could be reused broadly in other
-
Can Robots Feel Music and Authentic Empathy?
By
–
Could a robot be moved by music? Could robots have authentic feelings and empathy? A short talk I gave at @DCameron84
's AI-empathy conference which includes my "origin story" in working at the intersection of emotions and robots. https://
youtu.be/oJehPSPYpi4?si
=c0rvd8AMsS7tAjbY&t=676
… -
NULL BITMAP: Building Databases with Log-Centric Architecture
By
–
NULL BITMAP Builds a Database #1: The Log is Literally the Database https://
bit.ly/4cxTGsM #AI #MachineLearning #DeepLearning #LLMs #DataScience -
New Llama Models Enable Synthetic Data Generation and Model Distillation
By
–
We believe these new Llama models will ignite new applications and modeling paradigms, including synthetic data generation and model distillation. With this release, we’ve updated our license to allow developers to use outputs from Llama models — including 405B — to improve
-
Llama 3.1 Models Feature 16x Larger Context Window
By
–
Hearing feedback from the community, all of our new Llama 3.1 models deliver a 16x increase in context window going all the way up to 128K tokens. This enables new uses in long text summarization, coding and more. We took a careful approach to the balance of data used in
-
Technical Details on New Llama 3.1 Models Released
By
–
More technical details on the new Llama 3.1 models we released today.
-
Private Benchmarks Show 405B Model Ranks Top 4
By
–
Ojito que estos benchmarks son más interesantes por ser privados y por tanto no estar filtrados en los datasets de los modelos. El modelo 405B fácil en el top 4
-
Zuckerberg’s Open-Source AI Strategy: Detailed Analysis Required
By
–
someone needs to publish a detailed exegesis of Zuck's post at https://
about.fb.com/news/2024/07/o
pen-source-ai-is-the-path-forward/
… ASAP