Hah, and even further, with pretrained models available, anything more than finetuning feels like a pain
@rasbt
-
Schmidhuber’s 1991 Alternative to RNNs: The Origins of Linear Transformers
By
–
Hah, yeah, little known fun fact: Schmidhuber proposed an alternative to RNNs back in 1991, which is now called "linear Transformers" or "Transformers with linearized self-attention" via more recent papers. Summarized it here: https://
magazine.sebastianraschka.com/p/why-the-orig
inal-transformer-figure
… -
Transformer’s Birthday: Attention Mechanisms and RNN Evolution
By
–
Happy birthday, transformer! An awesome summary @DrJimFan
! Also interesting to think about why we needed attention for RNNs (before transformers) in the first place. Since we can't translate word-by-word, we needed a RNN encoder-decoder setup. But then, it's hard to remember. -
Attention Mechanism Origins Traced Back to 1991 Schmidhuber
By
–
Oh, and sorry, @DrJimFan
, even the 2014 paper is not the original attention paper. It goes actually back to 1991 (via @SchmidhuberAI
) in https://
semanticscholar.org/paper/Learning
-to-Control-Fast-Weight-Memories-An-to-Schmidhuber/bc22e87a26d020215afe91c751e5bdaddd8e4922
… I've summarized it here: https://
magazine.sebastianraschka.com/p/why-the-orig
inal-transformer-figure
… -
Deep Learning Lecture: From RNNs to Transformer Architecture
By
–
Link to the lecture in case someone's curious: https://
lightning.ai/pages/courses/
deep-learning-fundamentals/unit-8.0-natural-language-processing-and-large-language-models/8.4-from-rnns-to-the-transformer-architecture/
… -
Bahdanau Attention as Essential Foundation for Transformer Learning
By
–
Yup! I always start my book chapters, lectures, etc with Bahdanau attention paper. I think it's a great intro & motivation for developing transformers. Without it, it would feel like sth is missing.
-
Comparison of BoW and BERT Models on Same Dataset
By
–
Yes, the two links above. It's both on the same dataset. One is a BoW model, one is a BERT model.
-
Adapter v2 and Video-LLaMA Models Available
By
–
There's both, adapter v2 and e.g., Video-LLaMA https://
github.com/DAMO-NLP-SG/Vi
deo-LLaMA
… -
Comparison of BOW and BERT Performance on Text Classification Tasks
By
–
The results for the 150 texts are from a paper (unpublished); from a collaboration so I can't share, yet. But I do also have examples on GitHub with a different dataset (from my course) here: 88% acc BOW: https://
github.com/Lightning-AI/d
l-fundamentals/blob/main/unit08-large-language-models/8.2-bag-of-words/8.2-part2-bag-of-words-classifier.ipynb
… 92.6% acc BERT: https://
github.com/Lightning-AI/d
l-fundamentals/blob/main/unit08-large-language-models/8.7-distilbert-finetuning/part3_distilbert-finetuning-whole.ipynb
… -
Open Source Models Now Support Multiple Media Types Beyond Images
By
–
Yeah, in the demo they showed a version that supported images. But still, afaik it was only images whereas there are now open source models that also support audio, video, etc.