If you liked my NN+Gzip article last week (
https://
magazine.sebastianraschka.com/p/large-langua
ge-models-and-nearest
…), you'll probably love @abhi9u & @alepiad 's follow-up. This one looks at the NN+Gzip method from a compression algo perspective comparing Gzip, Huffman coding, LZ77/LZ4, and Bzip2: https://
codeconfessions.substack.com/p/lz77-is-all-
you-need/
…
@rasbt
-
NN+Gzip Follow-up: Compression Algorithms Comparison Study
By
–
-
Model Performance Comparison with 11B T5 Parameters
By
–
Probably about the same as the 11B T5 model
-
Installation and requirements setup for deep learning workshop
By
–
Maybe this installation + requirements.txt from my recent DL workshop is useful here: https://
github.com/rasbt/scipy202
3-deeplearning/tree/main/00-2_python-libraries-for-workshop
… -
Power User PyTorch Nightly and Lightning GitHub Installation
By
–
As a "power user" I usually install the PyTorch nightly and the lightning from the GitHub branch .
-
Fine-tuning smaller models for specialized tasks with custom data
By
–
Yes, it's more impressive. But if you are a small company that specializes on a specific task and has some custom data, I'd say that finetuning a smaller (e.g. 1.5 B) model is the way to go.
-
Fine-tuning Custom LLMs vs Few-Shot Prompting Performance Comparison
By
–
2/3 For fairness: The DeBERTa-1.5B model was likely finetuned on the training data whereas Llama 2 was used via few-shot prompting. In that case, it highlights once more that finetuning custom LLMs remains worthwhile.
-
Reference Papers for Llama 2 and DeBERTa Models
By
–
3/3
Here are the papers for reference: Llama 2: Open Foundation and Fine-Tuned Chat Models: https://
arxiv.org/abs/2307.09288 DeBERTa: Decoding-enhanced BERT with Disentangled Attention: https://
arxiv.org/abs/2006.03654 -
DeBERTa-1.5B Outperforms Llama 2 on BoolQ Classification Task
By
–
Interestingly, DeBERTa-1.5B (and encoder-only model) beats Llama 2 on BoolQ, which is a nice example that encoders still outperform large decoders on classification task. For fairness: The DeBERTa-1.5B model was likely finetuned on the training data 1/3
-
RLHF and Encoder Modules in Decoder Architectures
By
–
Sure I don’t disagree, but since all these were before RLHF finetuned decoder architectures, I am curious whether it’s still necessary to have an encoder module. LLaMA-Adapter would be an example of adding img2txt capabilities w/o cross attention.
-
Fancy Index Lookups in Embedding Layers and Batch Matrix Operations
By
–
*fancy index lookups in embedding layers followed by fancy batch matrix-matrix products