(And yes, that's not a typo, you can use LLaMA-Adapter for Falcon :P)
@rasbt
-
LoRA vs LLaMA-Adapter: Runtime Performance Comparison
By
–
I'd suggest LoRA or LLaMA-Adapter. Either one works fine and has almost exactly the same runtime. Will upload a benchmark later today.
-
Parameter-Efficient Fine-Tuning of Pretrained Large Language Models
By
–
Now, if we talk about parameter-efficient finetuning of a pretrained LLM, that's a different story.
-
GPU Training Complexity: Weeks on Hundreds vs CPU Decades
By
–
I'd stick to practically impossible; it already takes weeks/months on 100s of GPUs. It would probably take decades on CPUs 😛
-
LLMs limitation acknowledged in discussion
By
–
Yeah, agreed. But unfortunately, that wouldn't be possible with LLMs.
-
CUDA Deterministic Mode Limitations Across Different Hardware
By
–
I think this is not possible because of CUDA requirements; you can force deterministic mode but in my experience results will still differ across hardware.
-
Free NLP and Large Language Models Course with Recommended Writings
By
–
I have a free intro course on Natural Language Processing and Large Language Models here that might be useful: https://
lightning.ai/pages/courses/
deep-learning-fundamentals/unit-8.0-natural-language-processing-and-large-language-models/
… I also have a bunch of writings here (Including a list of papers I recommend): https://
magazine.sebastianraschka.com -
Upcoming Update on Current Project Coming This Week
By
–
Yes, coincidentally, I am currently working on it. Stay (fine)tuned for an update on that this week!
-
Mathematical Definition of Nut Cracking Using Set Theory
By
–
Haha true. Let N be the set of all nuts, and let
C : N → P(P(N)) be the function for cracking a nut, where P(N) is the power set of N. Then, For every nut n in N, if n is cracked via C(n)), the cardinality of the resulting set of nutshells |C(n)| > 2. -
LLaMA-Adapter and LoRA: Multimodal Input Advantages
By
–
Good question. I'd say LLaMA-Adapter and LoRA are both good methods. The advantage of LLaMA-Adapter though is that you can also embed multimodal inputs (images and others)