Extraction of structured data from unstructured text is a hugely impactful (tho less sexy) use case for LLMs Excited to have revamped docs on this use case
LLMS
-
RLHF and Encoder Modules in Decoder Architectures
By
–
Sure I don’t disagree, but since all these were before RLHF finetuned decoder architectures, I am curious whether it’s still necessary to have an encoder module. LLaMA-Adapter would be an example of adding img2txt capabilities w/o cross attention.
-
ChatGPT won’t revolutionize programming as much as feared
By
–
@mikeloukides on how things like ChatGPT will change programming, but not as much as we may think https://
oreilly.com/radar/fearing-
the-wrong-thing/
… -
Curiosity about empirical results in NeurIPS LLM efficiency challenge
By
–
Oh yeah, that makes intuitive sense. I am just curious about how this works out empirically. (The NeurIPS LLM efficiency challenge permits encoder, encoder-decoder, and decoder LLMs, so will will probably have to wait until the leaderboard is finalized haha)
-
Decoder-Only Architectures for Vision-Language Models
By
–
That makes sense. But you could also use a decoder-only architecture (with embedded image tokens as part of the input, as in LLaMA-Adapter, for example). (* This uses an encoder for the tokens, but it's still a decoder-only architecture due to the lack of cross-attention)
-
Neural Networks with Gzip Compression for Language Models
By
–
NN + Gzip because we got bored waiting https://
magazine.sebastianraschka.com/p/large-langua
ge-models-and-nearest
… -
Comparison of decoder models with ChatGPT and GPT-4 after RLHF
By
–
! I’m mainly curious how it compared to decoders like ChatGPT or GPT-4 after RHLF
-
Comparing Decoder-Only Models to Encoder-Decoder Architectures for Translation
By
–
The original transformer is an encoder-decoder arch for translation. T5 is a great encoder-encoder that’s pretty good at translation. ChatGPT / GPT-4 is a decoder-only that’s pretty good at translation too. How does it compare to encoder-decoder architectures of similar size?
-
Clarification on inference versus pretraining data requirements
By
–
You mean during inference or regarding amount of data needed for pretraining?
-
Encoder-Decoder vs Decoder-Only Architecture Training Efficiency
By
–
You mean, assuming that both an encoder-decoder and a decoder-only architecture, the encoder-decoder is easier to train because you make better use of the data due to the masking pretraining tasks?