To make ALiBi extrapolate better, we applied position interpolation – an idea popularized in models that use RoPE. Position interpolation scales the input to fit in the context length used during training, thus generating more stable results.

→ View original post on X — @cerebras