Hah, yeah, little known fun fact: Schmidhuber proposed an alternative to RNNs back in 1991, which is now called "linear Transformers" or "Transformers with linearized self-attention" via more recent papers. Summarized it here: https://
magazine.sebastianraschka.com/p/why-the-orig
inal-transformer-figure
…
MACHINE LEARNING
-
Schmidhuber’s 1991 Alternative to RNNs: The Origins of Linear Transformers
By
–
-
Transformer’s Birthday: Attention Mechanisms and RNN Evolution
By
–
Happy birthday, transformer! An awesome summary @DrJimFan
! Also interesting to think about why we needed attention for RNNs (before transformers) in the first place. Since we can't translate word-by-word, we needed a RNN encoder-decoder setup. But then, it's hard to remember. -
Augmenting Language Models with Long-Term Memory
By
–
Augmenting Language Models with Long-Term Memory paper page: https://
huggingface.co/papers/2306.07
174
… Existing large language models (LLMs) can only afford fix-sized inputs due to the input length limit, preventing them from utilizing rich long-context information from past inputs. To address -
Weakly Supervised Information Extraction from Handwritten Document Images
By
–
Weakly supervised information extraction from inscrutable handwritten document images paper page: https://
arxiv.org/abs/2306.06823 State-of-the-art information extraction methods are limited by OCR errors. They work well for printed text in form-like documents, but unstructured, -
High-Fidelity Audio Compression with Improved RVQGAN
By
–
High-Fidelity Audio Compression with Improved RVQGAN paper page: https://
huggingface.co/papers/2306.06
546
… Language models have been successfully used to model natural signals, such as images, speech, and music. A key component of these models is a high quality neural compression model that can -
Enhancing Deep Reinforcement Learning with Quality Diversity Algorithms
By
–
Improving Deep Reinforcement Learning via Quality Diversity, Open-Ended and AI-Generating Algorithms https://
bit.ly/3WjxeMq #AI #MachineLearning #DeepLearning #LLMs #DataScience -
The Impact of ChatGPT on Artificial Intelligence
By
–
The impact of #ChatGPT #ArtificialIntelligence #GenerativeAI #NLP #DataScientist
— Helen Yu (@YuHelenYu) 13 juin 2023
TK @BrivaelLp pic.twitter.com/pGGAyS6RxnThe impact of #ChatGPT #ArtificialIntelligence #GenerativeAI #NLP #DataScientist TK @brivael
-
Training T5 Model on Language Modeling Objective
By
–
Training t5 on lm objective for 100k more steps. There are lm adapted ckpts of t5 on the internet iirc.
-
Tokenizer Effects on Math Tasks: T5 vs LM Adaptation
By
–
Math tasks could be due to c4 sentencepiece/tokenizer though. T5 can't open end generate that well due to span corruption objective but LM adaptation or ul2 fixes this
-
Meta Scales Speech Technology to 1,100+ Languages Globally
By
–
Our work on the Massively Multilingual Speech (MMS) project has scaled speech-to-text & text-to-speech to support 1,100+ languages + trained new language identification models that can identify 4,000+ languages!
— AI at Meta (@AIatMeta) 12 juin 2023
More info & access to pretrained models ⬇️Our work on the Massively Multilingual Speech (MMS) project has scaled speech-to-text & text-to-speech to support 1,100+ languages + trained new language identification models that can identify 4,000+ languages! More info & access to pretrained models