It will take time, but as both a wish and a prediction for 2025, I hope to see more interesting approaches to developing open-weight (o1 and o3-like) reasoning models through improved post-training recipes, data, and inference compute.
@rasbt
-
Tokenizer Implementation Limitations in Educational Materials
By
–
Thanks for sharing those, it's really nice seeing the chapters through the lens of a reader! Regarding the chapter 2 nit. It’s more of a shortcoming of the tokenizer implementation (it’s a very simple one for educational purposes). The reason why I copied it as is from the
-
DeepSeek-V3 achieves superior performance with 11x less compute than Llama
By
–
> E.g. Llama 3 405B used 30.8M GPU-hours, while DeepSeek-V3 looks to be a stronger model at only 2.8M GPU-hours (~11X less compute). Super interesting! And DeepSeek was trained in H800’s which are probably also a tad (or noticeably?) slower than Meta’s H100’s.
-
Voice assistants prove invaluable during medical emergency in Japan
By
–
Never liked voice assistants and only used Siri when I absolutely had to, like when driving. But last month when I ended up in an ER in Japan that was the only way to communicate with the doctors and will forever appreciate it (not ChatGPT but some voice translation app)
-
RoPE and GQA Implementation in GPT to Llama Conversion Guide
By
–
Glad to hear that you've been enjoying it! Regarding RoPE and GQA, I've added some bonus material here a few months back where I implement it from scratch in the GPT -> Llama conversion guide: https://
github.com/rasbt/LLMs-fro
m-scratch/tree/main/ch05/07_gpt_to_llama
…
I was planning to add more, but unfortunately I am currently -
Final Chapter and Bonus Content Highlights for Readers
By
–
Hope that was fun (personally, writing the last chapter was my fav). Hope the bonus content came in handy as well! If you had to prioritize, I'd check out the DPO one and the GPT to Llama conversion guide!
-
Llama 3 Training Challenges: Outages and Checkpoint Recovery
By
–
That's true. But looking at the Llama 3 report, there's still a lot of work going into dealing with outages, recovering checkpoints during training, etc. If I recall correctly, they had like 500 interruptions. It's really hard to train a model on that scale.
-
GPU costs and hidden expenses in AI model training
By
–
100%. And even if they didn't own the GPUs, they would get discounts. But on the flipside, it doesn't include costs for failed runs, hyperparameter tuning, and researchers and engineer salaries etc.
-
GPU Rental Costs and DeepSeek V3 Technical Report
By
–
Of course, this is the sticker price of rental GPUs. Lots of bigger companies have their in-house clusters or discount deals with cloud providers. But still…
Btw the full report is here: https://
github.com/deepseek-ai/De
epSeek-V3/blob/main/DeepSeek_V3.pdf
… -
LLM Pretraining Costs Calculation Based on DeepSeek-v3
By
–
An updated back-of-the-envelope calculation of LLM pretraining costs based on the just-released DeepSeek-v3 report.
And that doesn't even account for hyperparameter tuning, failed runs, or personnel costs. It really makes me appreciate the value of openly shared model weights!