We hope these results are a step towards a mechanistic theory of memorization. There are many open questions, such as understanding the loss spike, or what happens when only a subset of the data is repeated.
AI
-
Model Capacity and Double-Descent: Strategy Transitions in ML
By
–
Models struggle to transition between these strategies, as exhibited by a spike in test loss. This spike moves to larger datasets as one increases model capacity. This is a clear signature of double-descent, a phenomenon that is now well-known in the ML literature.
-
Superposition in Neural Networks: Memorization vs Feature Learning
By
–
For small training sets, models use superposition to memorize more data points than the two available neurons. For large training sets, models learn features in superposition, as observed in our previous work, allowing the model to generalize. https://
transformer-circuits.pub/2022/toy_model
/index.html
… -
Understanding Deep Learning Overfitting Through Mechanistic Analysis
By
–
We have little mechanistic understanding of how deep learning models overfit to their training data, despite it being a central problem. Here we extend our previous work on toy models to shed light on how models generalize beyond their training data. https://
transformer-circuits.pub/2023/toy-doubl
e-descent/index.html
… -
Superposition Strategy: How Neural Networks Embed Features in Hidden Space
By
–
Our prior work showed that these toy models use a strategy called “superposition” to learn more features than available neurons. Here we observe how training data points, as well as features, are embedded in the hidden space.
-
GroqWare Suite: Foundation of Software-Defined Hardware Approach
By
–
The foundation of our software-defined hardware approach is the GroqWare™ suite. Learn about our product offerings here: https://
groq.link/groqware -
LLMs Will Get Better This Year, Solving Trivial Problems
By
–
Prototypes are easy, production is hard. However, LLMs will get a **lot** better this year. Expect many unsolved problems to become trivial. But not all.
-
The fundamental difference between software and non-software AI development trajectories
By
–
Yes, that’s the real explanation for it — sci-fi AI was imagined as software for consumer hardware, so once it eats the tail of its development it goes FOOM. The fact it isn’t software changes everything.
-
Unified Communications AI Productivity Hybrid Workers Equality
By
–
Diary Date 🙂 Join @Mitel #CMO @VenkatNagaswamy & I for #LinkedIn #LiveStream on #tech & #people aspects of #UnifiedCommunications – Ensuring #equality of #Productivity for #Hybrid workers Jan 12th http://
bit.ly/CommsUnified #telco #5G #AI #MitelPartner #CES2023 #innovation -
LifeOxa Wearable Shirt Sensor Guides Focus and Calm
By
–
@LifeOxa is a sensor built into your t-shirt that measures your breathing. It is the first wearable that guides you into a state of focus and calm, avoiding stress and other health issues We are at CES 2023! Visit us at booth #61433 to try Oxa #Oxa #ces2023 #health #sleep