visiting some mathematician friends who work on language models and i'm struck by how different the lifestyle is. their workspaces are mountains of papers surrounded by endless blackboards. they talk of mutual info, topological theory, PAC bounds. i am completely covered in chalk
RESEARCH
-
Cerebras Sparsity Optimization for Foundation Model Training
By
–
(10/n) Contact us to learn more about Cerebras and how sparsity can make training your next foundation model orders of magnitude more efficient. Shoutout to the amazing software, machine learning, and performance team members who’ve played an instrumental role in developing
-
Modular Library Advances Sparse Machine Learning Research
By
–
(9/n) Our library is modular and extensible, enabling users to define algorithms without complex rewrites. It is already pivotal in supporting in-house research on sparsity, demonstrated through our works: *Sparse-IFT: https://
openreview.net/pdf?id=iP4WcJ4
EX0
…, and https://
proceedings.mlr.press/v216/thangaras
a23a.html
… * -
Flexible Sparsity Distributions for Advanced Model Optimization
By
–
(7/n) We provide a lot of flexibility, including custom sparsity distributions, schedules for sparsity, and other richer features such as grouped tensor views and tie-breaking mechanisms.
-
RigL Dynamic Weight Sparsity Algorithm Implementation PyTorch
By
–
(8/n) These allow us to implement popular dynamic weight sparsity algorithms such as Rigging the Lottery (RigL) from @GoogleDeepMind
. Enabling these algorithms for any existing PyTorch model in our ModelZoo is as simple as making a few changes in our configuration file. -
PyTorch Sparse Optimizers for Efficient Model Training
By
–
(6/n) We rely on PyTorch’s optimizer interface to design all our sparse algorithms as special optimizers, which operate on a sparse view of the model parameters and gradients during training. We use a wrapper over existing PyTorch optimizers, which enables efficient
-
Library Enables Dense and Sparse AI Training Workloads
By
–
(4/n) Our library has been co-designed from the ground up to run both dense and #sparse training workloads. This enables practitioners to focus on what they do best: push the boundaries of AI research.
-
Cerebras CS-2 Accelerates Foundation Model Training Through Sparsity
By
–
(5/n) Our library is hardware agnostic, but combined with the Cerebras CS-2's unique ability to accelerate unstructured #sparsity, it unleashes unparalleled efficiency in training foundation models.
-
Sparsity Unlocks New ML Training Efficiency Dimension
By
–
(3/n) Sparsity helps unlock a new dimension of efficiency beyond model architectures for training, enabling control of #computing performance in the ML practitioner's hand. Sparse models also achieve better scaling but are difficult to accelerate. Today's #deeplearning libraries
-
Cerebras Releases PyTorch Sparsity Library for ML Researchers
By
–
(1/n) Excited to announce the "Cerebras PyTorch Sparsity Library," which democratizes access to #sparse training for all #ML researchers and developers. Read more here: https://
cerebras.net/blog/sparsity-
made-easy-introducing-the-cerebras-pytorch-sparsity-library
…