“RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts” RoPE is the default way LLMs track position, but this paper proves it breaks down as context gets long. Attention becomes close to random at telling near from far, and can even assign the same score to different