(1/2) On November 14th and 15th, stop by the @psc_live booth at #SC23 for an informational session. Join us from 2:00 – 2:45 pm MST for this session to learn how you can develop kernels and deliver scientific computing research using the Cerebras SDK on Neocortex
@cerebras
-
WANT Workshop Trains Neural Networks at Scale
By
–
(2/2) The WANT workshop provides researchers with the tools to train neural networks at scale. It will provide an interactive platform for researchers and practitioners to delve into the latest advancements in neural network training Learn more here: https://
want-ai-hpc.github.io -
Sparse Transformations Maximize Neural Network Training Efficiency
By
–
(1/2) We are excited to share that our paper "Sparse Iso-FLOP Transformations for Maximizing Training Efficiency" has been accepted by the Workshop on Advancing Neural Network Training (WANT) at #neurips2023! Read our blog for more information: https://
cerebras.net/blog/can-spars
ity-make-ai-models-more-accurate
… -
State-of-the-Art LLM Training Techniques and Learnings
By
–
(2/2) Our paper details learnings from training state-of-the-art LLMs, including: -Rotary and ALiBi position embeddings
-Swish-gated linear unit (SwiGLU)
-Overtraining on many Tokens-Per-Parameter
-LR Decay Ratio
-Maximal update parameterization (muP) https://
arxiv.org/abs/2309.11568 -
Cerebras Accelerates 175B LLM Training with PyTorch Single Device
By
–
Cerebras presented at the PyTorch Developer Conference and covered how we accelerate Large Language Model (175B+) training using PyTorch, Torch-MLIR, and the simplicity of single device training without a single torch.distributed instruction Watch here:
-
Cerebras Co-Founder Presents AI Hardware Innovations at OktoberTech
By
–
Cerebras Co-Founder JP Fricker had a great time presenting at OktoberTech 2023! This annual event by @Infineon Technologies brought together some of the brightest minds in the tech industry to share their perspectives on the latest trends and innovations. #Digitalization
-
ALiBi Models Performance Extension Without Fine-Tuning
By
–
Unlike existing techniques – this method requires no additional fine tuning. By setting a few parameters in your HuggingFace config.json, you can instantly extend the performance of ALiBi models such as BTLM-3B-8K by ~2x: https://
huggingface.co/cerebras/btlm-
3b-8k-base#during-inference-without-fine-tuning
… -
Position Interpolation Extends ALiBi Model Context From 8K to 16K
By
–
We show that position interpolation works just as well in the context of ALiBi – models trained with 8K of context can now extrapolate up to 16K.
-
Position Interpolation Improves ALiBi Model Extrapolation
By
–
To make ALiBi extrapolate better, we applied position interpolation – an idea popularized in models that use RoPE. Position interpolation scales the input to fit in the context length used during training, thus generating more stable results.
-
ALiBi Context Extrapolation Limits in Production Language Models
By
–
ALiBi promises fast, long extrapolation but in production models only extrapolate 10-20% longer than its training context. This is why ALiBi models such as BTLM-3B-8K and MPT-7B-8K are trained on a mix of 2K and 8K contexts – ALiBi on its own cannot extrapolate from 2K to 8K.