The book is GPT-2 based, which is more beginner friendly. The Llama 3 code is bonus material. (It makes sense to implement GPT-2 first and than convert that to Llama 3 imho.) Llama 3 is dense, not an MoE though.

→ View original post on X — @rasbt