Rotary positional embeddings (aka RoPE) have been a recent cornerstone of modern LLM implementations since it supports flexible sequence lengths. In this paper, researchers propose Position Interpolation to increase RoPE-based context window sizes to 32,768 tokens 2/3
@rasbt
-
Extending Transformer LLMs Context Window via Positional Interpolation
By
–
In the last couple of days, we talked a lot about extending the context window of transformer LLMs. Here's one more: "Extending Context Window of Large Language Models via Positional Interpolation" 1/3
-
Zero-Sum Performance Trade-offs in Attention Mechanisms
By
–
Yeah but my guess is it’s a zero sum issue. Like if you fixing the performance in the middle, you will probably have to sacrifice performance elsewhere. Otherwise if you pay attention to everything equally, you’ll lose the advantage of attention in a way.
-
Appreciation for Different Content Direction from Standard LLM Topics
By
–
Glad to hear that you liked this slight departure from the usual LLM content!
-
Hyena Convolutional LLMs: Efficient Alternative to Transformers
By
–
Hyena convolutional LLMs look interesting. Not necessarily better but matches the performance more or less. It’s different but more efficient. We will see if this will get more interest vs continuing the transformer route.
-
Recommending a non-speculative arXiv research paper
By
–
One relatively good one is https://
arxiv.org/abs/2212.08073. I don’t find it that speculative tbh. -
The Food Chain of AI, Machine Learning, and Computer Science
By
–
AI eats ML eats CS as the food chain goes
-
Roberta-Large and Decoder-Style LLMs Comparison
By
–
I would consider Roberta-Large (see above) an LLM. But you are also welcome to swap that with decoder-style LLMs. I have a BLOOM (GPT-3 equivalent) example here:
-
Positional Embeddings and Learned Query-Key Weights in Transformers
By
–
That's true, but you still have positional embeddings, and Q and K weights that are learned
-
Fine-tuning RoBERTa for Sentiment Classification to 95% Accuracy
By
–
Then, you can boost that performance (on the same movie review sentiment classification dataset) to ~95% finetuning model = AutoModelForSequenceClassification.from_pretrained( "siebert/sentiment-roberta-large-english", num_labels=2)