Good writeup. Here's an interesting fact about paged attention: It isn't free at the kernel level. As expected, the non-contiguous memory reads add roughly 20-26% overhead per attention kernel call. But the system-wide gain (2-4x throughput) overshadows that because you can

→ View original post on X — @akshay_pachaar