The "imaginary" part of RoPE contains valuable information for long-range dependencies RoPE++ reintroduces it in heads with slower decay curves that better capture distant tokens We provide implementations with one-command scripts for both the original codebase and Nanochat!

→ View original post on X — @askalphaxiv