Hmm I don’t think they use sliding window attn ? Where did you see that?

→ View original post on X — @rasbt