This is the training degeneracy that @jiawzhao is talking about if you only do identity initialization and not Hadamard for rectangular weight matrices
MACHINE LEARNING
-
AGI Parameter Count vs Brain Synapses: Current Models Scale
By
–
A common view is that human level AGI will require a parameter count in the order of magnitude of the brain’s 100 trillion synapses. The large language models and image generators are only about 1/1000 of that, but they already contain more information than a single human
-
Training Transformers at Scale for Scientific Applications
By
–
Transformers and other large language models have shown impressive capabilities as foundation models for domains such as NLP and CV. Join Andy Hock, our VP of Product, at #SC22, as he discusses new algorithms, software, and hardware for training transformers at scale for science
-
Cool Applications Of Artificial Intelligence In Modern Technology
By
–
Cool Applications Of #ArtificialIntelligence
Via @ingliguori #BigData #Analytics #DataScience #AI #IoT #IIoT #PyTorch #Python #TensorFlow #Java #JavaScript #CloudComputing #DevCommunity #DataScientist #Programming #DigitalTransformation #Cloud #MachineLearning #100DaysofCode -
Detecting Major AI Breakthroughs Before They Happen
By
–
a thing about research i didn't get before openai: frequently before a big idea gets figured it out, multiple teams can sort of detect it on radar through the fog. you get an idea of where it's going to be and the rough shape far before anyone actually lays eyes on it.
-
Emergent Capabilities in Large-Scale Language Models Explained
By
–
As language models increase in scale they sometimes exhibit various useful emergent capabilities. In today’s post, we explore this phenomenon to better understand its dependence on model properties and its potential impact in applications. Read more at https://
goo.gle/3hvJMAb -
Hadamard Transform Essential for Dimension Changes and Degeneracy
By
–
The dimension changes need Hadamard transform and without it you have bad degeneracy. We discuss this in detail
-
Potential Use Cases of Artificial Intelligence in Healthcare
By
–
#Infographic: Potential use cases of #AI in #healthcare
via @ingliguori #HealthTech #ArtificialIntelligence #Robotics #MedTech #DigitalTransformation #Cloud #MachineLearning #BigData #DX #Analytics #IIoT #DataScience #IoT #IoTPL #DEVCommunity -
H2O World Dallas Conference November 17 Machine Learning Event
By
–
Join us in person or virtually next week on November 17 at H2O World Dallas! The makers behind http://
H2O.ai will share the latest technology advancements, use cases, and best practices in machine learning. Attend H2O World Dallas: https://
hopin.com/events/h2o-wor
ld-dallas-b3f1dc1d-e6dc-4955-a2f8-2d22e7aeb5c2/registration?utm_campaign=h2o-world-dallas-2022&utm_source=twitter
… -
Larger AI Models Show Increased Bias Demonstration
By
–
It looks like the bigger the model, the harder it gets for it not to demonstrate biases. Very cool stuff from @mathemakitten
! https://
huggingface.co/blog/zero-shot
-eval-on-the-hub
…