CODE

A little spoiler: 1. @EverMind's MSA paper has been selected for publication by AlphaXiv, and it's getting great traffic 2. MSA's Inference code will be open sourced this week github.com/EverMind-AI/MSA — alphaXiv (@askalphaxiv) Scaling Attention to 100M context!? Memory Sparse Attention introduces an idea where instead of rereading an entire 100M-token entry, it learns to jump straight into the relevant memories and reason from them end-to-end. More specifically, it first encodes documents into compressed memory slots, then for each question it uses a learned router to score which chunks are actually relevant, pulls only the top few, and runs normal attention over that tiny assembled context. So the model's compute grows with "how much it needs to look at" not "how much memory exists". This retrieval step is trained jointly with answer generation, so memory lookup is part of the model itself, and can decouple memory capacity from reasoning cost. [Translated from EN to English]

→ View original post on X — @elliotchen100, 2026-04-01 03:05 UTC