Unlike existing techniques – this method requires no additional fine tuning. By setting a few parameters in your HuggingFace config.json, you can instantly extend the performance of ALiBi models such as BTLM-3B-8K by ~2x: https://
huggingface.co/cerebras/btlm-
3b-8k-base#during-inference-without-fine-tuning
…
LLMS
-
ALiBi Models Performance Extension Without Fine-Tuning
By
–
-
Position Interpolation Extends ALiBi Model Context From 8K to 16K
By
–
We show that position interpolation works just as well in the context of ALiBi – models trained with 8K of context can now extrapolate up to 16K.
-
Position Interpolation Improves ALiBi Model Extrapolation
By
–
To make ALiBi extrapolate better, we applied position interpolation – an idea popularized in models that use RoPE. Position interpolation scales the input to fit in the context length used during training, thus generating more stable results.
-
ALiBi Context Extrapolation Limits in Production Language Models
By
–
ALiBi promises fast, long extrapolation but in production models only extrapolate 10-20% longer than its training context. This is why ALiBi models such as BTLM-3B-8K and MPT-7B-8K are trained on a mix of 2K and 8K contexts – ALiBi on its own cannot extrapolate from 2K to 8K.
-
Position Interpolation Doubles Context Length for ALiBi Models
By
–
Paper drop: Position Interpolation Improves ALiBi Extrapolation We found a simple method to 2x the context length of models that use ALiBi. This lets models like BTLM-3B-8K and MPT-7B-8K run high quality inference at up to 16K with no additional fine tuning.
-
Auto-Regressive LLMs Limited Reasoning Cannot Be Fixed by Scaling
By
–
Anyone who thinks Auto-Regressive LLMs are getting close to human-level AI, or merely need to be scaled up to get there, *must* read this. AR-LLMs have very limited reasoning and planning abilities.
This will not be fixed by making them bigger and training them on more data. -
GPT-4 DALLE3 prompt interpretation differences explained
By
–
And as pointed out, if you are on the web-version of GPT4/DALLE3 you can indeed click on the images and get your full prompts, but these are still not the ones you wrote in the first place. But the GPT4 interpretation based on your initial ask.
-
Extracting GPT-4 System Instructions: Challenges and Insights
By
–
Here is how the pre-cursor instructions are, I've yet been able to get GPT4 to exactly spill the beans on the system instructions.
-
Emergent Abilities in AI: Training’s Hidden Role Revealed
By
–
I’ve often thought this. The “emergent” abilities seem less emergent when you know, for example, that skilled programmers have spent thousands of hours training the model.