"Fast KV Compaction via Attention Matching" The current problem is that long-context LLMs would choke on KV cache memory, so systems either drop tokens or write lossy summaries. As a solution, the paper shows you can shrink an entire prefix to ~50x fewer KV slots by solving

→ View original post on X — @askalphaxiv