Glad to hear that you've been enjoying it! Regarding RoPE and GQA, I've added some bonus material here a few months back where I implement it from scratch in the GPT -> Llama conversion guide: https://
github.com/rasbt/LLMs-fro
m-scratch/tree/main/ch05/07_gpt_to_llama
…
I was planning to add more, but unfortunately I am currently

→ View original post on X — @rasbt