Just catching up with the ReLoRA paper (
https://
arxiv.org/abs/2307.05695) that explores whether LoRA can be used for pretraining LLMs (vs finetuning). Looks promising! Caveat: they pretrained models up to 350 M parameters (the smallest Llama model is 7 B parameters, for comparison)
@rasbt
-
ReLoRA Paper: Exploring LoRA for LLM Pretraining Beyond Finetuning
By
–
-
Removing dataset to ensure fair neural network method comparison
By
–
Thanks. And yeah, good call removing that dataset because especially NN methods would have an unfair advantage there
-
Gzip and Neural Networks Paper Remains Fun and Engaging
By
–
It's refreshing to see how much fun we (still) get out of the Gzip + NN paper. It was provocative, intuitive, and easily hackable: the perfect storm for a rainy weekend.
-
Dataset Query and RoBERTa-Large Performance on IMDB Reviews
By
–
Which dataset have you used in this figure? If you want to boost it a few more percentage points, RoBERTa-Large is worth trying. Here are some results on the IMDB review dataset: https://
github.com/Lightning-AI/d
l-fundamentals/discussions/41
… -
Bag-of-Words Models Outperform Gzip in Text Classification
By
–
Oh yeah, Bag-of-Words models are more favorite simple baseline for text classification. In my IMDB experiments, BoW ranked ~10% above the Gzip method (but 6% below LLM embeddings)
-
Limited demand for on-premise generative vision models customization
By
–
Not saying there’s no need for generative vision models, but I guess fewer companies have the need to run it on prem or customize it vs using a SOTA off-the-shelf one. And even fewer companies even have the training data required for meaningful customization.
-
Custom LLMs Use Cases: Privacy-Sensitive Drafting and Domain Knowledge
By
–
There may also be more usecases for custom LLMs atm: – assistants for all sorts of drafting (don’t want to post sensitive materials into eg ChatGPT)
– Q&A for legal, medical, or financial data & docs
– customer chatbots that have some domain knowledge wrt to the company products -
Rethinking Academic Requirements Beyond Paper Publications
By
–
No papers no PhD. But hey, maybe we can go back to the earlier days when a solid thesis (in academia or industry) was enough, and students weren’t pressured to get >= 3 top-tier conference papers accepted before graduation.
-
Need Multimodal ChatGPT and Bard for Screenshot Analysis
By
–
We need multimodal ChatGPT or Bard so we can copy-paste the screenshot and ask
-
Building Things: The Key to Learning Deep Learning Fundamentals
By
–
The secret sauce for getting into modern deep learning & AI. For me "building things" has been a powerful motivator with respect to learning how things work under the hood (later).