Starting on using PoSE to extend LLaMA 3's context window, with @winglian
's help. This one was really fun to work on. Eval loss started insanely high, let's see how far down we can get it.
Extending LLaMA 3 Context Window with PoSE Method
By
–
By
–
Starting on using PoSE to extend LLaMA 3's context window, with @winglian
's help. This one was really fun to work on. Eval loss started insanely high, let's see how far down we can get it.