Did you know that the embedding layer can contain 63% of total model parameters? In this talk, I present unique challenges of small models from architecture (don't build giant embedding layers) to post-training (how to fix doom looping) ↓ Slides in the comments ↓

→ View original post on X — @maximelabonne,