Congrats to Sean Lie, Cerebras Chief Hardware Architect and Co-Founder on being featured on the AI 100 list! Sean led the effort on Weight-Streaming, our breakthrough architecture that lets many CS-2s work as one networked AI supercomputer. https://
businessinsider.com/the-ai-100-202
3-the-people-who-make-ai-intelligent-2023-10
…
@cerebras
-
Sean Lie Recognized on AI 100 List for Weight-Streaming Architecture
By
–
-
BTLM Achieves 7B Performance in 3B Parameter Model
By
–
In Davis Blalock's latest arXiv roundup, BTLM is mentioned as achieving 7B parameter performance in a 3B parameter model. Davis mentions, "I’m not sure I’ve seen any other non-GPU hardware vendors do this with a model of this size and quality." Read here: https://
dblalock.substack.com/i/136702777/bt
lm-b-k-b-parameter-performance-in-a-b-parameter-model
… -
Cerebras Prioritizes India for AI Hardware Development
By
–
In a recent interview with India's @EconomicTimes
, Cerebras CEO Andrew Feldman said he categorically considers India as a priority for Cerebras, citing the country's tremendous engineering talent, top universities, and a growing AI ecosystem. Read here: -
Cerebras Nominated for HPC Industry Awards and New Products
By
–
(2/2) Please vote for Cerebras and our partners in the following categories: -Best Use of HPC in Industry
-Best HPC Collaboration
-Top Supercomputing Achievement
-Top 5 New Products or Technologies to Watch
-Top 5 Vendors to Watch -
Cerebras Nominated for HPCwire Readers’ Choice Awards
By
–
(1/2) ICYMI, Cerebras and our partners have been nominated for the @HPCwire Readers’ Choice Awards in various categories! Please vote for Cerebras and our partners in the following categories by October 1 by clicking here: https://
hpcwire.com/2023-hpcwire-r
eaders-choice-awards-voting-is-open/
… -
BTLM-3B Outperforms Larger 7B Models in Long Context
By
–
The paper explores long context performance in detail. We found BTLM-3B-8K outperforms other 7B-8K models despite being trained with less than a fifth the pretraining compute and less than half the size.
-
ALiBi Enables Context Extrapolation Beyond Training Length
By
–
ALiBi allows our model trained on 8192 context length to extrapolate well up to 9216 tokens out of the box. We also show that ALiBi models only achieve near-perfect extrapolation when they are severely undertrained (<1 tokens/parameter).
-
BTLM: World’s Most Accurate 3B Parameter Open Source Model
By
–
BTLM is the world’s most accurate 3B parameter model, with performance that rivals many 7B models. It’s fully open source and has been downloaded >1M times.
-
LLM Training Techniques: 2.86x Compute Efficiency Gains
By
–
There are so many LLM training tricks but which ones should you use? We quantify the effects of techniques like SwiGLU, ALiBi, etc and show how using them in conjunction can achieve the same loss with 2.86x less pretraining compute or 1.74x less parameters.
-
BTLM-3B-8K: Distilling SOTA LLM Training Recipe
By
–
We just dropped the BTLM-3B-8K paper on arXiv! It distills our recipe for training SOTA LLMs:
– Extensively deduplicated dataset (SlimPajama)
– Hyperparameter search using muP
– Variable sequence length training + ALiBi
– Aggressive LR decay https://
arxiv.org/abs/2309.11568