(2/2) Our paper details learnings from training state-of-the-art LLMs, including: -Rotary and ALiBi position embeddings
-Swish-gated linear unit (SwiGLU)
-Overtraining on many Tokens-Per-Parameter
-LR Decay Ratio
-Maximal update parameterization (muP) https://
arxiv.org/abs/2309.11568
State-of-the-Art LLM Training Techniques and Learnings
By
–
