Yeah, there is a silver lining. But I think it gets a bit muddled since most base models are trained on CoT data.
@rasbt
-
Optimal RL Balance in Training Reasoning Models
By
–
Another really interesting paper from my 2025 bookmarked papers: On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (https://arxiv.org/abs/2512.07783). In short, RL is most effective when applied to data that is neither too close to nor too far from the
-
Text Diffusion as Alternative to Standard LLMs
By
–
I wrote a bit about it here https://
magazine.sebastianraschka.com/p/beyond-stand
ard-llms
… but in short, I don't think there's a real viable alternative at the moment.
– Text diffusion is interesting as a lower-cost option (Google is planning to launch Gemini Diffusion some time as alt. to their lowest cost Flash -
Increasing Context Length and Model Capacity: Expensive Trade-offs
By
–
Either
a) you increase the context length and/or let the model crawl through stored data
b) you start of with a model that has large enough capacity and periodically update the weights
Both options are expensive, so I am not sure they actually want this right now. -
Experimenting with Olmo 3 and DeepSeek V3.2 GRPO tweaks
By
–
Oh yeah you probably will. I am currently running tons of experiments to decide which of the Olmo 3 and DeepSeek V3.2 GRPO tweak I should add in the final version.
No side objective really except I'd do an honest evaluation to make sure that the model actually learns well. -
Pre-trained Base Models Lacked Instruction-Following Capabilities
By
–
Yeah, I guess that’s right. They were probably only pre-trained base models at that time and didn’t have good instruction-following abilities.
-
Google’s Risk Aversion: Why OpenAI Launched Chatbots First
By
–
They had the Meena and Lamda chatbots, which came out in 2020 and 2021 (
https://
en.wikipedia.org/wiki/LaMDA).
I think a startup would have launched them, but Google didn't because of risk.
They needed someone else to test the waters first. I think that is why OpenAI was first. -
Google’s Different Direction versus OpenAI’s Approach
By
–
Yes exactly. So, I think it's not "OpenAI ran with it" but "Google ran into a different direction with it"
-
Encoder vs Decoder: Google and OpenAI’s Architectural Rivalry
By
–
I am not sure "OpenAI ran with it" is entirely correct. I remember there was quite some rivalry between Google's encoder approach and OpenAI's push for decoder-style models.
Google tried to make encoder-style models work for many years, since the original architecture. -
DeepSeek V3.2 sparse attention mod overshadows throughput focus announcement
By
–
Based on the announcement post it seems the focus is token/sec throughput, cost, and latency. Not sure if that’s because the modeling performance wasn’t exceeding other models or if it was the original plan.
But even then there is now DeepSeek V3.2 with the sparse attention mod.