Traditional pre-training had diminishing returns (which is what the “scaling law”predicted anyway) The fact that reasoners were developed at exactly the moment where pre-training faltered is exactly the pattern of how Moore’s Law works: new technique appear to maintain the trend
LLMS
-
OpenAI doubles GPT-5 rate limits for ChatGPT Plus users
By
–
GPT-5 rollout updates: *We are going to double GPT-5 rate limits for ChatGPT Plus users as we finish rollout. *We will let Plus users choose to continue to use 4o. We will watch usage as we think about how long to offer legacy models for. *GPT-5 will seem smarter starting
-
Extracting Training Data Directly from Large Language Models
By
–
FUTURE WORK – direct extraction we're working on directly extracting training data from models using RL and other methods. we'll be presenting our first work on this in COLM, and expect more in this space we may be able to directly extract data from the 120b model.. one day
-
Describing Text Distribution Differences Between Language Models
By
–
FUTURE WORK – describing differences @ZhongRuiqi has some incredible work on methods for describing the difference between two text distributions *in natural language* we could compare outputs of 20b to the 120b model, or LLAMA, or GPT-5…
-
Understanding Constant Codeswitching in Language Models
By
–
what are some explanations for constant codeswitching? 1. OpenAI has figured out RL. the models no longer speak english
2. data corruption issues via OCR or synthetic training
3. somehow i forced the model to output too many tokens and they gradually shift out of distribution -
AI Models Generate Creative Screenplay Examples
By
–
there are a small number of creative outputs interspersed throughout here's one example where the model starts writing a sketch for a norwegian screenplay
-
Multilingual Reasoning Chains in Neural Language Models
By
–
what you can't see from the map is many of the chains start in English but slowly descend into Neuralese the reasoning chains happily alternate between Arabic, Russian, Thai, Korean, Chinese, and Ukrainian. then usually make their way back to English (but not always)
-
Model Hallucinates Domino Problem Repeatedly in Token Loop
By
–
and it truly is a tortured model. here the model hallucinates a programming problem about dominos and attempts to solve it, spending over 30,000 tokens in the process completely unprompted, the model generated and tried to solve this domino problem over 5,000 separate times
-
Embedded Generations: AI Model Capabilities in Math and Code
By
–
here's a map of the embedded generations the model loves math and code. i prompt with nothing and yet it always reasons. it just talks about math and code, and mostly in English math – probability, ML, PDEs, topology, diffeq
code – agentic software, competitive programming, -
GPT-OSS Training Data Analysis: Bizarre Results Revealed
By
–
curious about the training data of OpenAI's new gpt-oss models? i was too. so i generated 10M examples from gpt-oss-20b, ran some analysis, and the results were… pretty bizarre time for a deep dive