The NeurIPS 2023 LLM efficiency challenge of course 🙂 https://
llm-efficiency-challenge.github.io Just recommend your Ulmfit paper with @seb_ruder to the students I am collaborating with the other day. Would be looking forward to the solutions you come up with for this one
@rasbt
-
NeurIPS 2023 LLM Efficiency Challenge and ULMFiT Recommendation
By
–
-
OpenLLaMA: An Open Source Large Language Model Alternative
By
–
You might like OpenLLaMA then: https://
github.com/openlm-researc
h/open_llama
… -
Llama-2 Base Model and Dataset Considerations Table
By
–
They use Llama-2 as a base model, so that should be okay. I am not a lawyer and can't give advice on the dataset aspect but here is a table from the paper that might be helpful
-
Theory Behind Q, K, V Matrices in Language Models
By
–
Not sure whether there is any theory behind it vs just empirical observation that it works well. Maybe an intuition is that the Q, K, V matrices work well for language in general, and you don't want to screw them up. Whereas the other ones are more like the extraction params.
-
Dolly 15k Support Merged for NeurIPS LLM Efficiency Challenge
By
–
If you are working on the NeurIPS LLM Efficiency Challenge, Dolly 15k support is now merged: https://
github.com/Lightning-AI/l
it-gpt/pull/430
… python scripts/prepare_dolly.py
python finetune/lora.py –data_dir data/dolly/ -
Platypus: Open-Source LLM Achieves Top Leaderboard Performance
By
–
Platypus, a new open-source LLM at the top of the leaderboard: https://
arxiv.org/abs/2308.07317 Key points are
1) a curated dataset: removing similar & duplicate questions
2) finetuning and merging Low Rank Approximation (LoRA) modules: focusing on the non-attention modules -
Machine Learning as Assembling Furniture from Existing Pieces
By
–
Sometimes ML feels like trying to assemble new furniture from pieces of existing furniture 😀
-
Personal fork of open-source LLM code repository
By
–
And a personal fork of an open-source LLM code repo 🙂
-
Adding Falcon to the List, Removing Bloom from Recommendations
By
–
I'd definitely add Falcon to the list (and remove Bloom, which was pre-2023)