The original transformer is an encoder-decoder arch for translation. T5 is a great encoder-encoder that’s pretty good at translation. ChatGPT / GPT-4 is a decoder-only that’s pretty good at translation too. How does it compare to encoder-decoder architectures of similar size?
@rasbt
-
Clarification on inference versus pretraining data requirements
By
–
You mean during inference or regarding amount of data needed for pretraining?
-
Encoder-Decoder vs Decoder-Only Architecture Training Efficiency
By
–
You mean, assuming that both an encoder-decoder and a decoder-only architecture, the encoder-decoder is easier to train because you make better use of the data due to the masking pretraining tasks?
-
Encoder vs Decoder LLMs: Benefits for Seq2Seq Tasks
By
–
Use cases for encoder LLMs (classification) & decoder LLMs (chatbots) are obvious. Seq2seq tasks like translation, where it makes sense to have access to the whole input, is where it gets interesting.
Re encoder-decoder LLMs (eg T5): Is there still a benefit of using an encoder? -
Restarting Cosine Schedule at Epochs or Checkpoints
By
–
That's actually a good point. But it would be quite trivial to restart the cosine schedule after each ~epoch or checkpoint
-
Multi-epoch training effectiveness for Vision Transformers
By
–
A counter-argument is that multi-epoch training works pretty well for ViTs.
-
Scaling LLM Performance: Token-Crisis Solutions and Diminishing Returns
By
–
They actually might be doing this already but yeah, probably diminishing returns.
Some insights in the "To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis" paper: https://
arxiv.org/abs/2305.13230 -
JAX Deep Learning API: Keras Backend and Framework Choices
By
–
The problem with JAX always on the deep learning API side. Imho, it needed something like Keras. Nice that it’s now a backend! Btw, do use pure JAX in Keras or one of the intermediate APIs (like Flax)?
-
RetentionNet and other model architecture examples mentioned
By
–
You are welcome! And yes, those two I mentioned above would be examples of that. Also RentionNet but that didn’t fit within the tweet limit
-
Transformers Can Be Applied to Tabular Non-Time Series Data
By
–
Yup there are transformers for tabular non-time series data