Just updated the Big LLM Architecture Comparison article…
…it grew quite a bit since the initial version in July 2025, more than doubled! https://
magazine.sebastianraschka.com/p/the-big-llm-
architecture-comparison
…
@rasbt
-
Sebastian Raschka Updates Comprehensive LLM Architecture Comparison Article
By
–
-
Architectural Variations in Modern Language Models
By
–
Yes, they are all relatively related, but usually they have a unique tweak like RMSNorm placement, sliding window, in this case MLA, etc.
I wrote about it in more detail here: -
Attention Mechanism Ratios in Language Models: Olmo vs Gemma
By
–
Do you remember if they used it in a 3:1 ratio (like Olmo 3) or 5:1 ratio (like Gemma 3) or just pure sliding window attention?
-
Decision Made Before Evaluation, Based on Training Loss
By
–
They made the choice even before evals. I.e., just looking at the training loss
-
MHA vs GQA Independence in Olmo 3 Models
By
–
Hard to say, but I think it may be independent of MHA vs GQA. Also, Olmo 3 7B uses MHA, and 32B uses GQA.
-
Olmo 2 Ablation Study on Loss Spikes and Normalization Techniques
By
–
In their Olmo 2 report they had an ablation study showing it reduces the loss spikes during training (but they also included QK norm, so it's hard to say how much of that reduction is due to QK norm and their post norm flavor).
Maybe best of both words is to do both like Gemma -
Olmo Models Highlighted for Transparency and Detailed Technical Reports
By
–
Olmo models are always a highlight due to them being fully transparent and their nice, detailed technical reports. I am sure I'll talk more about the interesting training-related aspects from that 100-pager in the upcoming days and weeks.
In the meantime, here's the side-by-side -
Appreciation for Olmo 3 spelling standardization and keyboard ease
By
–
Just started reading… thanks for canonicalizing the Olmo 3 spelling! (Rolls much easier off the keyboard than OLMo 2 where I was never quite sure which exact letters had to be capitalized :D)
-
The computational cost challenge of large k values in pruning
By
–
The only caveat: 32 is still crazy large. And if you use the k=100 before pruning, that's 100x more compute!
-
Self-Consistency Outperforms Best-of-N Approaches in LLM Decoding
By
–
I see! Thanks. I remember now. Not in that form, i.e., not directly (still running too many other experiments). But a few thoughts: 1. Self-consistency does seem to be still ahead of any best-of-N approach. This is based on my experience but also a recent paper I saw: 18 Apr