We show that position interpolation works just as well in the context of ALiBi – models trained with 8K of context can now extrapolate up to 16K.
Position Interpolation Extends ALiBi Model Context From 8K to 16K
By
–
By
–
We show that position interpolation works just as well in the context of ALiBi – models trained with 8K of context can now extrapolate up to 16K.