LLMs rely heavily on transformers, which make use of an "attention mechanism" for modeling interactions among inputs. But this attention mechanism is inefficient and thus difficult to scale to longer contexts. Because of this, researchers have been experimenting w/"linear

→ View original post on X — @mit_csail