Sonnet 4.5 hits 82% on SWE-bench with test-time compute. We're adding another dot to what's been a pretty clean exponential toward saturating SWE-bench as a benchmark. At this rate we'll need new evals soon, but it's still a useful signal that the capability curve is holding.
MACHINE LEARNING
-
AI Impact on Programming Careers Reassurance Note
By
–
More on my blog, including this hopefully reassuring note for anyone afraid of the impact this will have on their career as a programmer https://
simonwillison.net/2025/Sep/29/ar
min-ronacher-90/
… -
AI-Generated Code in Infrastructure: Armin’s Credible 90% Achievement
By
–
Most of the "90% of code written by AI" claims come from vendors selling AI tools and lack credibility as a result Armin (creator of Flask, Jinja, Click) is different – when he says 90% of a new infrastructure project he's building was AI generated it's worth paying attention
-
Hyperparameter Changes for Muon Optimization and Optimizer Comparison
By
–
What hyperparameters changed when optimizing for muon? @clashluke has tried several new optimizers on our code and not reliably beaten Adamw.
-
24 YOLO Streams: Sub-1J/Frame Edge AI Efficiency
By
–
24 #YOLO streams. 1 edge card. Real-time. Independent tests show sub-1 J/frame efficiency + consistent performance across YOLOv5 → YOLOv8. Multi-stream vision at the edge is finally practical: Retail. Security. Robotics. Is this where you imagined things going?
-
Symbolic Verifiers vs LLM Judges for Verification Tasks
By
–
What do you mean with set of prompts? A LLM judge with a rubric? (That’s a different approach; covered in the appendix).
The reason for a symbolic verifier is that it is symbolic. There is no nondeterminism, bias, etc. Doesn’t work for everything, but eg for math there is no -
Frequency of car brand citations in AI EV recommendations
By
–
Hello, the number of times car brands are cited when asking for EV recommendations to different AI models.
-
DeepSeek Achieves 50x Attention Efficiency Breakthrough
By
–
DeepSeek casually unlocked 50x attention efficiency in ~1 year > MLA is ~5.6x faster than MHA
> DSA is 9x faster than MLA never doubted you, you big beautiful whale -
What Should Replace MMLU for AI Model Evaluation?
By
–
Final update! MMLU is saturated and has become (rightfully) less popular. What should replace it? – Other knowledge evals like GPQA, MMLU-Pro
– Code evals like LiveCodeBench
– Agentic evals like BFCL
– Other?