I am really excited for the research community to develop (more) efficient methods for finetuning LLMs.
And I hope you find this competition as useful and exciting as I do! Link to the competition here:
@rasbt
-
Excitement for Efficient LLM Fine-tuning Research Methods
By
–
-
NeurIPS 2023 LLM Efficiency Challenge Starter Guide Released
By
–
The NeurIPS 2023 LLM Efficiency Challenge is a super exciting opportunity for developing & benchmarking new research directions for param-efficient LLMs. If you are looking for something to tinker with this weekend, I just put together a Starter Guide: https://
lightning.ai/pages/communit
y/tutorial/neurips2023-llm-efficiency-guide/
… -
Question about paper submission phrasing for real conferences
By
–
would we say "we submitted a paper to the ICML" if it was a real conference? 😛
-
arXiv Papers Compared to GitHub Code Repositories
By
–
“The paper is on the arXiv” would be like “the code is on the GitHub”
-
ReLoRA Paper: Exploring LoRA for LLM Pretraining Beyond Finetuning
By
–
Just catching up with the ReLoRA paper (
https://
arxiv.org/abs/2307.05695) that explores whether LoRA can be used for pretraining LLMs (vs finetuning). Looks promising! Caveat: they pretrained models up to 350 M parameters (the smallest Llama model is 7 B parameters, for comparison) -
Removing dataset to ensure fair neural network method comparison
By
–
Thanks. And yeah, good call removing that dataset because especially NN methods would have an unfair advantage there
-
Gzip and Neural Networks Paper Remains Fun and Engaging
By
–
It's refreshing to see how much fun we (still) get out of the Gzip + NN paper. It was provocative, intuitive, and easily hackable: the perfect storm for a rainy weekend.
-
Dataset Query and RoBERTa-Large Performance on IMDB Reviews
By
–
Which dataset have you used in this figure? If you want to boost it a few more percentage points, RoBERTa-Large is worth trying. Here are some results on the IMDB review dataset: https://
github.com/Lightning-AI/d
l-fundamentals/discussions/41
… -
Bag-of-Words Models Outperform Gzip in Text Classification
By
–
Oh yeah, Bag-of-Words models are more favorite simple baseline for text classification. In my IMDB experiments, BoW ranked ~10% above the Gzip method (but 6% below LLM embeddings)
-
Limited demand for on-premise generative vision models customization
By
–
Not saying there’s no need for generative vision models, but I guess fewer companies have the need to run it on prem or customize it vs using a SOTA off-the-shelf one. And even fewer companies even have the training data required for meaningful customization.