Isn’t Stability AI exclusively using AWS?
@rasbt
-
Google Search Ads Evolution: Subtler Ad Integration Strategy 2024
By
–
For those who complain about Google Search ads: Ads in 2024 will be become a lot more subtle …
-
Focusing on Improving Modeling Performance Over Computational Performance
By
–
*the focus here is on improving modeling performance, not computational performance.
-
Humble approach to long context lengths with minimal finetuning
By
–
This sounds a bit more humble than the recent 1M or 1B context lengths claims. But this requires only minimal (1000 steps) finetuning. And it allows long document summarization using LLaMA 7B and 65B models, for example. Link to the paper here: https://
arxiv.org/abs/2306.15595 3/3 -
Position Interpolation Extends RoPE Context Windows to 32K Tokens
By
–
Rotary positional embeddings (aka RoPE) have been a recent cornerstone of modern LLM implementations since it supports flexible sequence lengths. In this paper, researchers propose Position Interpolation to increase RoPE-based context window sizes to 32,768 tokens 2/3
-
Extending Transformer LLMs Context Window via Positional Interpolation
By
–
In the last couple of days, we talked a lot about extending the context window of transformer LLMs. Here's one more: "Extending Context Window of Large Language Models via Positional Interpolation" 1/3
-
Zero-Sum Performance Trade-offs in Attention Mechanisms
By
–
Yeah but my guess is it’s a zero sum issue. Like if you fixing the performance in the middle, you will probably have to sacrifice performance elsewhere. Otherwise if you pay attention to everything equally, you’ll lose the advantage of attention in a way.
-
Appreciation for Different Content Direction from Standard LLM Topics
By
–
Glad to hear that you liked this slight departure from the usual LLM content!
-
Hyena Convolutional LLMs: Efficient Alternative to Transformers
By
–
Hyena convolutional LLMs look interesting. Not necessarily better but matches the performance more or less. It’s different but more efficient. We will see if this will get more interest vs continuing the transformer route.
-
Recommending a non-speculative arXiv research paper
By
–
One relatively good one is https://
arxiv.org/abs/2212.08073. I don’t find it that speculative tbh.