The performance of LoRA-trained models is usually pretty competitive. Sometimes even better than fully finetuned (prob due to less overfitting).
(Table from https://
arxiv.org/abs/2106.09685)
OPEN SOURCE
-
LoRA-trained models show competitive performance with less overfitting
By
–
-
LLaMA-Adapter v2 Adds Trainable RMSNorm and Bias Parameters
By
–
Good question. Yes, because LLaMA-Adapter v2 is basically LLaMA-Adapter but it also has trainable RMSNorm parameters and trainable bias units (in FC layers)
-
Efficient Falcon Finetuning with LoRA on Single GPU
By
–
Have been heads-down porting LoRA (low-rank-adaptation) to finetune Falcon more efficiently and ran some performance benchmarks. My longer write-up here: https://
lightning.ai/pages/communit
y/finetuning-falcon-efficiently/
… Long story short: you can finetune Falcon in 1 h on a 52k dataset using a single GPU with 16 GB RAM. -
LLaMA-Adapter Compatible with Falcon Models
By
–
(And yes, that's not a typo, you can use LLaMA-Adapter for Falcon :P)
-
LoRA vs LLaMA-Adapter: Runtime Performance Comparison
By
–
I'd suggest LoRA or LLaMA-Adapter. Either one works fine and has almost exactly the same runtime. Will upload a benchmark later today.
-
Embedding Models: Ada-002 vs Open Source Competition
By
–
Embedding models are one of the most important parts of model development and deployment. You have to encode all your data in the first place. But while GPT-4 is well ahead of open source competition in LLMs, ada-002 didn't necessarily have that level of a commanding lead.
-
HuggingFace Transformers Integration Strategy with Descript
By
–
Hey Ishaan, saw that your DM's are closed, can you DM me, I'd love to discuss potential HF transformers integrations and a deeper strategy with Descript and HF.
Given how much both orgs love open source, it sounds like a good next step! -
Claim Paper Pages on Hugging Face Without Model Dataset or Space
By
–
and you can claim papers even if there is no model, dataset or space for it: https://
huggingface.co/docs/hub/paper
-pages#can-i-have-a-paper-page-even-if-i-have-no-modeldatasetspace
… -
Finetuning Open Source Models for Tool Use Webinar
By
–
Don't let all the OpenAI `functions` distract you from the fact we've got an amazing webinar starting in an hour focused on open source models with @nomic_ai (GPT4ALL) and @MosaicML Can we finetune those models for tool use? Let's find out!
-
How to Claim Authorship of Your Paper on Hugging Face Hub
By
–
docs on how to claim your paper here: https://
huggingface.co/docs/hub/paper
-pages#claiming-authorship-to-a-paper
…