“TriAttention: Efficient Long Reasoning with Trigonometric KV Compression”
— alphaXiv (@askalphaxiv) 8 avril 2026
Most KV-cache compression methods guess what to keep by looking at recent attention.
But this paper argues that the signal is unstable because RoPE keeps rotating queries with position, so what looks… pic.twitter.com/GrEYc0gZ2b
“TriAttention: Efficient Long Reasoning with Trigonometric KV Compression” Most KV-cache compression methods guess what to keep by looking at recent attention. But this paper argues that the signal is unstable because RoPE keeps rotating queries with position, so what looks unimportant now may matter later. So they proposed TriAttention, which looks in the pre-RoPE space and finds that many heads have stable Q/K centers. That lets it predict which token distances a head is likely to retrieve, and compress the KV cache using that structure rather than noisy recent attention. This shift from "keeping what was attended recently” to “keeping what this head is likely to need later” Empirically, it matches full attention on AIME25 with 2.5x higher throughput or 10.7x less KV memory.