China trained a model called DeepSeek-v3 that’s better than OpenAI GPT 4o with a small fraction of the budget and open sourced it. Meanwhile, a bipartisan bill is in the works in America to impose severe restrictions on open sourcing frontier models.
LLMS
-
DeepSeek-V3 achieves superior performance with 11x less compute than Llama
By
–
> E.g. Llama 3 405B used 30.8M GPU-hours, while DeepSeek-V3 looks to be a stronger model at only 2.8M GPU-hours (~11X less compute). Super interesting! And DeepSeek was trained in H800’s which are probably also a tad (or noticeably?) slower than Meta’s H100’s.
-
RoPE and GQA Implementation in GPT to Llama Conversion Guide
By
–
Glad to hear that you've been enjoying it! Regarding RoPE and GQA, I've added some bonus material here a few months back where I implement it from scratch in the GPT -> Llama conversion guide: https://
github.com/rasbt/LLMs-fro
m-scratch/tree/main/ch05/07_gpt_to_llama
…
I was planning to add more, but unfortunately I am currently -
Final Chapter and Bonus Content Highlights for Readers
By
–
Hope that was fun (personally, writing the last chapter was my fav). Hope the bonus content came in handy as well! If you had to prioritize, I'd check out the DPO one and the GPT to Llama conversion guide!
-
Dense vs Mixture of Experts: Technical Insights and Public Sources
By
–
Very interesting, are there any more intuition on dense vs MoE? Also hate to be that guy.. but is there a public source for this (to test okay if not)
-
Wanting mixed text/audio interaction in ChatGPT on mobile
By
–
I feel like mixed text/audio output is still missing. On mobile it’s weird ChatGPT fills the screen with nothing vs. even a transcript. Like, I want it to be able to ask for a small code example, see it on my phone, then say out loud “great, now in Rust” and it happens silently.
-
ChatGPT Down, Introducing the New Google Gemini
By
–
ChatGPT is down today. Well timed, I'm talking about Google Gemini today on the channel. I buried Google too quickly — Here is the new Gemini! https://youtu.be/fK9lRG3onEE
-
Why models don’t know their own release version
By
–
A model can’t know its own release version because it’s usually only decided that a trained model *should* be a release version once it’s fully post-trained, red-teamed, etc. Human demonstrations are necessarily older. Claude I assume would learn it in RLAIF.
-
Grok 2 and 3 Release Expectations from xAI
By
–
All eyes on @xai HF org page – expecting Grok 2 to be released as soon as Grok 3 is released! Hoping for a classic @elonmusk drop before the end of the year!
-
Comment on system prompt and post-training practicality
By
–
Yeah sorry, was unclear this is coming from the system prompt — the reason I ask this is because it’s impractical to teach it in post-training. Added a comment to the thread: