They had the Meena and Lamda chatbots, which came out in 2020 and 2021 (
https://
en.wikipedia.org/wiki/LaMDA).
I think a startup would have launched them, but Google didn't because of risk.
They needed someone else to test the waters first. I think that is why OpenAI was first.
@rasbt
-
Google’s Risk Aversion: Why OpenAI Launched Chatbots First
By
–
-
Google’s Different Direction versus OpenAI’s Approach
By
–
Yes exactly. So, I think it's not "OpenAI ran with it" but "Google ran into a different direction with it"
-
Encoder vs Decoder: Google and OpenAI’s Architectural Rivalry
By
–
I am not sure "OpenAI ran with it" is entirely correct. I remember there was quite some rivalry between Google's encoder approach and OpenAI's push for decoder-style models.
Google tried to make encoder-style models work for many years, since the original architecture. -
DeepSeek V3.2 sparse attention mod overshadows throughput focus announcement
By
–
Based on the announcement post it seems the focus is token/sec throughput, cost, and latency. Not sure if that’s because the modeling performance wasn’t exceeding other models or if it was the original plan.
But even then there is now DeepSeek V3.2 with the sparse attention mod. -
Sebastian Raschka Updates Comprehensive LLM Architecture Comparison Article
By
–
Just updated the Big LLM Architecture Comparison article…
…it grew quite a bit since the initial version in July 2025, more than doubled! https://
magazine.sebastianraschka.com/p/the-big-llm-
architecture-comparison
… -
Architectural Variations in Modern Language Models
By
–
Yes, they are all relatively related, but usually they have a unique tweak like RMSNorm placement, sliding window, in this case MLA, etc.
I wrote about it in more detail here: -
Attention Mechanism Ratios in Language Models: Olmo vs Gemma
By
–
Do you remember if they used it in a 3:1 ratio (like Olmo 3) or 5:1 ratio (like Gemma 3) or just pure sliding window attention?
-
Decision Made Before Evaluation, Based on Training Loss
By
–
They made the choice even before evals. I.e., just looking at the training loss
-
MHA vs GQA Independence in Olmo 3 Models
By
–
Hard to say, but I think it may be independent of MHA vs GQA. Also, Olmo 3 7B uses MHA, and 32B uses GQA.
-
Olmo 2 Ablation Study on Loss Spikes and Normalization Techniques
By
–
In their Olmo 2 report they had an ablation study showing it reduces the loss spikes during training (but they also included QK norm, so it's hard to say how much of that reduction is due to QK norm and their post norm flavor).
Maybe best of both words is to do both like Gemma