We recently announced the availablity of SlimPajama – an open-source, cleaned, and deduplicated version of RedPajama-1T. It is half the size and trains twice as fast and when upsampled, performs equal or better than RedPajama. See below for the dataset and preprocessing library
MACHINE LEARNING
-
askgpt: AI-Powered Chat Interface for Learning R Programming
By
–
Introducing `askgpt`: a chat interface that helps you to learn R! | Johannes B. Gruber https://
bit.ly/3Mp9GBc #AI #MachineLearning #DeepLearning #LLMs #DataScience -
Open Source Model Evaluation Results Exceed Public Competition
By
–
thanks for running the evals. indeed the numbers look great (and promising). yeah i think it's scalable and efficient. like your tweet suggested it's a pity there's no t5 or ul2 model >40B. otherwise i think it would beat every public open source model right now.
-
Advanced Computer Vision Course at UCF Center for Research
By
–
Advanced Computer Vision – UCF Center for Research in Computer Vision, Spring 2023 This is an excellent course on advanced computer vision. Covers recent developments in computer vision, topics like image generation and vision-language learning. I like that the course go
-
Encoder-Decoder vs Decoder-Only Models: T5 Paper Guide
By
–
Additional reading for people confused about encoder-decoder models vs decoder-only models: most wisdom is also from the 2019 T5 paper. it's the bible for NLP literally.
-
Attention Mechanism Origins Traced Back to 1991 Schmidhuber
By
–
Oh, and sorry, @DrJimFan
, even the 2014 paper is not the original attention paper. It goes actually back to 1991 (via @SchmidhuberAI
) in https://
semanticscholar.org/paper/Learning
-to-Control-Fast-Weight-Memories-An-to-Schmidhuber/bc22e87a26d020215afe91c751e5bdaddd8e4922
… I've summarized it here: https://
magazine.sebastianraschka.com/p/why-the-orig
inal-transformer-figure
… -
Deep Learning Lecture: From RNNs to Transformer Architecture
By
–
Link to the lecture in case someone's curious: https://
lightning.ai/pages/courses/
deep-learning-fundamentals/unit-8.0-natural-language-processing-and-large-language-models/8.4-from-rnns-to-the-transformer-architecture/
… -
Bahdanau Attention as Essential Foundation for Transformer Learning
By
–
Yup! I always start my book chapters, lectures, etc with Bahdanau attention paper. I think it's a great intro & motivation for developing transformers. Without it, it would feel like sth is missing.
-
Comparison of BoW and BERT Models on Same Dataset
By
–
Yes, the two links above. It's both on the same dataset. One is a BoW model, one is a BERT model.
-
The Promise and Pitfalls of Relying on Artificial Intelligence
By
–
The promise and pitfalls of relying on artificial intelligence
#AI #AIio #BigData #ML #NLU #Futureofwork @nigelwalsh @pbouillaud @PatrickGunz_CH @pierrepinna @RebekahRadice @Ronald_vanLoon @sbmeunier
@Shirastweet http://
ow.ly/Um1h30svHSJ