Oh yeah, that makes intuitive sense. I am just curious about how this works out empirically. (The NeurIPS LLM efficiency challenge permits encoder, encoder-decoder, and decoder LLMs, so will will probably have to wait until the leaderboard is finalized haha)
@rasbt
-
Decoder-Only Architectures for Vision-Language Models
By
–
That makes sense. But you could also use a decoder-only architecture (with embedded image tokens as part of the input, as in LLaMA-Adapter, for example). (* This uses an encoder for the tokens, but it's still a decoder-only architecture due to the lack of cross-attention)
-
Neural Networks with Gzip Compression for Language Models
By
–
NN + Gzip because we got bored waiting https://
magazine.sebastianraschka.com/p/large-langua
ge-models-and-nearest
… -
Foundation math textbooks and advanced ML AI resources
By
–
Depends on your background and interests. Math textbooks are usually a good foundation for most technical fields.
Shameless plug: if you are into ML and AI and want to pick up some more advanced concepts after an intro book, my ML Q & AI might be useful: -
NVIDIA GPU and CUDA Manual Confusion in Molecular Dynamics Lab
By
–
Can confirm: I was interning at a molecular dynamics lab ~10 years ago. NVIDIA gave us a Tesla GPU, which came with a ~500-page CUDA manual — no one knew what to do with it
-
Comparison of decoder models with ChatGPT and GPT-4 after RLHF
By
–
! I’m mainly curious how it compared to decoders like ChatGPT or GPT-4 after RHLF
-
Comparing Decoder-Only Models to Encoder-Decoder Architectures for Translation
By
–
The original transformer is an encoder-decoder arch for translation. T5 is a great encoder-encoder that’s pretty good at translation. ChatGPT / GPT-4 is a decoder-only that’s pretty good at translation too. How does it compare to encoder-decoder architectures of similar size?
-
Clarification on inference versus pretraining data requirements
By
–
You mean during inference or regarding amount of data needed for pretraining?
-
Encoder-Decoder vs Decoder-Only Architecture Training Efficiency
By
–
You mean, assuming that both an encoder-decoder and a decoder-only architecture, the encoder-decoder is easier to train because you make better use of the data due to the masking pretraining tasks?
-
Encoder vs Decoder LLMs: Benefits for Seq2Seq Tasks
By
–
Use cases for encoder LLMs (classification) & decoder LLMs (chatbots) are obvious. Seq2seq tasks like translation, where it makes sense to have access to the whole input, is where it gets interesting.
Re encoder-decoder LLMs (eg T5): Is there still a benefit of using an encoder?