Glad to hear that you've been enjoying it! Regarding RoPE and GQA, I've added some bonus material here a few months back where I implement it from scratch in the GPT -> Llama conversion guide: https://
github.com/rasbt/LLMs-fro
m-scratch/tree/main/ch05/07_gpt_to_llama
…
I was planning to add more, but unfortunately I am currently
@rasbt
-
RoPE and GQA Implementation in GPT to Llama Conversion Guide
By
–
-
Final Chapter and Bonus Content Highlights for Readers
By
–
Hope that was fun (personally, writing the last chapter was my fav). Hope the bonus content came in handy as well! If you had to prioritize, I'd check out the DPO one and the GPT to Llama conversion guide!
-
Llama 3 Training Challenges: Outages and Checkpoint Recovery
By
–
That's true. But looking at the Llama 3 report, there's still a lot of work going into dealing with outages, recovering checkpoints during training, etc. If I recall correctly, they had like 500 interruptions. It's really hard to train a model on that scale.
-
GPU costs and hidden expenses in AI model training
By
–
100%. And even if they didn't own the GPUs, they would get discounts. But on the flipside, it doesn't include costs for failed runs, hyperparameter tuning, and researchers and engineer salaries etc.
-
GPU Rental Costs and DeepSeek V3 Technical Report
By
–
Of course, this is the sticker price of rental GPUs. Lots of bigger companies have their in-house clusters or discount deals with cloud providers. But still…
Btw the full report is here: https://
github.com/deepseek-ai/De
epSeek-V3/blob/main/DeepSeek_V3.pdf
… -
LLM Pretraining Costs Calculation Based on DeepSeek-v3
By
–
An updated back-of-the-envelope calculation of LLM pretraining costs based on the just-released DeepSeek-v3 report.
And that doesn't even account for hyperparameter tuning, failed runs, or personnel costs. It really makes me appreciate the value of openly shared model weights! -
Author Notices ChatGPT Echoes Writing Habits from 2014 Book
By
–
Haha fun fact: I used to”delve” 2x in my 2014 Python Machine Learning book 10 years as well. ChatGPT must have learned that habit from my older books
-
Manual Feature Extraction and Count Vector Representations in ML
By
–
Oh I see what you mean. Here human experts don’t necessarily extract the features. What I meant was more like humans came up with the idea that count vectors are a good representation (vs inputting the raw sentence). Ie here with manual feature extraction I mean the
-
Naive Bayes Spam Filtering Feature Extraction Discussion
By
–
This is actually a fun discussion! Regarding naive Bayes for spam filtering, in my view, that's definitely manual feature extraction as you have to construct term frequency vectors as input to the model (i.e., bag of words). Alternatively, you could use embeddings like from
-
BERT masked token prediction and LLM next-word prediction clarification
By
–
Hope you are enjoying the book! And thanks for sharing your notes! Btw regarding your nits, I introduced the masked token prediction task of BERT in Fig 1.5 and even refer to BERT-like "LLMs", but you are right, I mostly equate LLMs with "next-word prediction" for simplicity's