While the discussion continues about whether GPT-5 meets expectations and also (as said in the Reddit AMA by Sam Altman) whether 256k context is enough or not – China enables up to 1m token context with their Qwen3 30b. China is delivering every day.
RESEARCH
-

Reasoners Emerge as Pre-Training Scaling Returns Diminish
By
–
Traditional pre-training had diminishing returns (which is what the “scaling law”predicted anyway) The fact that reasoners were developed at exactly the moment where pre-training faltered is exactly the pattern of how Moore’s Law works: new technique appear to maintain the trend
-
Extracting Training Data Directly from Large Language Models
By
–
FUTURE WORK – direct extraction we're working on directly extracting training data from models using RL and other methods. we'll be presenting our first work on this in COLM, and expect more in this space we may be able to directly extract data from the 120b model.. one day
-
Deduplicating Redundant AI-Generated Output Data
By
–
FUTURE WORK – deduplication even though i varied the random seed and used temperature, a lot of the outputs are highly redundant it would be prudent to deduplicate, i bet there are only 100k or fewer mostly-unique examples here
-
Describing Text Distribution Differences Between Language Models
By
–
FUTURE WORK – describing differences @ZhongRuiqi has some incredible work on methods for describing the difference between two text distributions *in natural language* we could compare outputs of 20b to the 120b model, or LLAMA, or GPT-5…
-

GPT-OSS 20B Samples Dataset Released on Hugging Face
By
–
if you want to try the data, here you go, it's on huggingface: http://
huggingface.co/datasets/jxm/g
pt-oss20b-samples
… let me know what you find! -

Model Unicode Skills and Physics Limitations Explored
By
–
i also learned a lot from this one. the model is *really* good at using unicode …but might be bad at physics. what in the world is a 'superhalo function'
-
Understanding Constant Codeswitching in Language Models
By
–
what are some explanations for constant codeswitching? 1. OpenAI has figured out RL. the models no longer speak english
2. data corruption issues via OCR or synthetic training
3. somehow i forced the model to output too many tokens and they gradually shift out of distribution -

OCR Conjecture: Evidence OpenAI Scanned Books for Training
By
–
the OCR conjecture: some examples include artifacts such as OCRV ROOT, which indicate the training data may have been reading between the lines: OpenAI is scanning books (for some reason the model loves mentioning how many deaf people live in Malaysia)
-
Multilingual Reasoning Chains in Neural Language Models
By
–
what you can't see from the map is many of the chains start in English but slowly descend into Neuralese the reasoning chains happily alternate between Arabic, Russian, Thai, Korean, Chinese, and Ukrainian. then usually make their way back to English (but not always)
