I really like the LLaMA-Adapter method! In a nutshell, it's about finetuning LLMs more efficiently. In a nutshell, it's adding a tunable prefix to the key and value tensors in the self-attention layers. Fun fact: it's not specific to LLaMA. Use it with any LLM!
LLMS
-
ChatGPT Impact on Research Integrity and Academic Culture
By
–
Very true! #chatgpt Sir Bashir M. Al-Hashimi بشير CBE FREng FRS (@hashimi_bashir) A lot is being written about ChatGPT and like impact on education, less so on the impact of such technology on research integrity/culture. Such technology accelerates the production of literature reviews, what about the production of unfounded research results? Views pls? — https://nitter.net/hashimi_bashir/status/1667974283878969347#m
→ View original post on X — @lailafaridoon, 2023-06-13 13:54 UTC
-
Five Ways to Use Generative AI at Work
By
–
Take Chat GPT to work . . .
5 ways to explore the use of generative AI at work https://
buff.ly/3NoBCqD
#ai #artificalintelligence #MachineLearning #DeepLearning #ChatGPT -
Llama.cpp Introduces Full CUDA GPU Acceleration Support
By
–
Llama.cpp: Full CUDA GPU Acceleration PR: https://
github.com/ggerganov/llam
a.cpp/pull/1827
… -
Transformers Learn Through Gradual Rank Increase
By
–
Transformers learn through gradual rank increase paper page: https://
huggingface.co/papers/2306.07
042
… identify incremental learning dynamics in transformers, where the difference between trained and initial weights progressively increases in rank. We rigorously prove this occurs under the -
Pretrained Models Make Fine-tuning the Only Practical Approach
By
–
Hah, and even further, with pretrained models available, anything more than finetuning feels like a pain
-
Schmidhuber’s 1991 Alternative to RNNs: The Origins of Linear Transformers
By
–
Hah, yeah, little known fun fact: Schmidhuber proposed an alternative to RNNs back in 1991, which is now called "linear Transformers" or "Transformers with linearized self-attention" via more recent papers. Summarized it here: https://
magazine.sebastianraschka.com/p/why-the-orig
inal-transformer-figure
… -
Transformer’s Birthday: Attention Mechanisms and RNN Evolution
By
–
Happy birthday, transformer! An awesome summary @DrJimFan
! Also interesting to think about why we needed attention for RNNs (before transformers) in the first place. Since we can't translate word-by-word, we needed a RNN encoder-decoder setup. But then, it's hard to remember. -
Augmenting Language Models with Long-Term Memory
By
–
Augmenting Language Models with Long-Term Memory paper page: https://
huggingface.co/papers/2306.07
174
… Existing large language models (LLMs) can only afford fix-sized inputs due to the input length limit, preventing them from utilizing rich long-context information from past inputs. To address -
High-Fidelity Audio Compression with Improved RVQGAN
By
–
High-Fidelity Audio Compression with Improved RVQGAN paper page: https://
huggingface.co/papers/2306.06
546
… Language models have been successfully used to model natural signals, such as images, speech, and music. A key component of these models is a high quality neural compression model that can