I think it wasn’t that computers were to slow, but rather getting yelled at by statisticians if the number of training parameters was bigger than the sample (training dataset) size
@rasbt
-
T5 and Flan-T5 Models Expected to Remain Highly Relevant
By
–
I'd say T5 and Flan-T5 are probably going to stay very relevant!
-
BERT and RoBERTa remain top models for classification tasks
By
–
BERT/RoBERTa are still among the best models for classification tasks. Even Twitter uses them here for their platform.
-
Distilled Models Remain Useful: GPT-3.5 and GPT-4 Examples
By
–
I do think that distilled models are still useful. I.e., GPT 3.5 in ChatGPT is a distilled model. And as far as I heard GPT 4.0 in ChatGPT is now also distilled.
-
PyTorch Distributed Training and Tensor Sharding with Fabric
By
–
None of these: just PyTorch with distributed training and tensor sharding. Optionally with CPU offloading for really big LLMs. I use Fabric as a convenient wrapper here.
-
AI Model Performance Less Impressive Than Data Memorization
By
–
Turns out, it’s even less impressive than memorizing the training data:
-
Understanding Encoder and Decoder LLMs: Dispelling Transformer Jargon
By
–
I often get requests to dispel some of the jargon behind transformers and LLMs! So here we go, my new article on "Understanding Encoder and Decoder LLMs":
-
Transformers in Diffusion Models for Image Processing Since 2022
By
–
Plus, since ~2022, Diffusion models also use transformers for the image processing. E.g.,
-
Text-to-Image Diffusion Models Use Transformers
By
–
Fun fact: common text-to-image Diffusion models actually use Transformers
-
DeepSpeed Issue Resolution in Finetuning Script
By
–
Btw it's not a DeepSpeed bug, more like something to address in the finetuning script! Sry for the confusion!