The performance of LoRA-trained models is usually pretty competitive. Sometimes even better than fully finetuned (prob due to less overfitting).
(Table from https://
arxiv.org/abs/2106.09685)
@rasbt
-
LoRA-trained models show competitive performance with less overfitting
By
–
-
Using 8 GPUs to reduce processing time to minutes
By
–
yes, you can of course use 8 GPUs to bring this down to a few minutes! 🙂
-
LLaMA-Adapter v2 Adds Trainable RMSNorm and Bias Parameters
By
–
Good question. Yes, because LLaMA-Adapter v2 is basically LLaMA-Adapter but it also has trainable RMSNorm parameters and trainable bias units (in FC layers)
-
LoRA vs Adapter: Performance and Memory Usage Comparison
By
–
Good question! On average: LoRA: 21.33 tokens/sec; Memory used: 14.59 GB
Adapter: 26.22 tokens/sec; Memory used: 14.59 GB -
Memory reduction: Full finetuning vs Adapter vs LoRA comparison
By
–
Btw, it's also a big reduction in memory: 16 gigs instead of 6 x 40 GB. Due to the reduced parameter counts in the backward pass: Full finetuning: 7,217,189,760
Adapter: 1,365,330
Adapter v2: 3,839,186
LoRA: 3,506,176 (I used LoRA rank of 16 to match Adapter v2 above.) -
Efficient Falcon Finetuning with LoRA on Single GPU
By
–
Have been heads-down porting LoRA (low-rank-adaptation) to finetune Falcon more efficiently and ran some performance benchmarks. My longer write-up here: https://
lightning.ai/pages/communit
y/finetuning-falcon-efficiently/
… Long story short: you can finetune Falcon in 1 h on a 52k dataset using a single GPU with 16 GB RAM. -
LLaMA-Adapter Compatible with Falcon Models
By
–
(And yes, that's not a typo, you can use LLaMA-Adapter for Falcon :P)
-
LoRA vs LLaMA-Adapter: Runtime Performance Comparison
By
–
I'd suggest LoRA or LLaMA-Adapter. Either one works fine and has almost exactly the same runtime. Will upload a benchmark later today.
-
Parameter-Efficient Fine-Tuning of Pretrained Large Language Models
By
–
Now, if we talk about parameter-efficient finetuning of a pretrained LLM, that's a different story.
-
GPU Training Complexity: Weeks on Hundreds vs CPU Decades
By
–
I'd stick to practically impossible; it already takes weeks/months on 100s of GPUs. It would probably take decades on CPUs 😛