It's 2025 and most content is still written for humans instead of LLMs. 99.9% of attention is about to be LLM attention, not human attention. E.g. 99% of libraries still have docs that basically render to some pretty .html static pages assuming a human will click through them.
@karpathy
-
Evaluation Crisis: MMLU Obsolete, Need Better AI Metrics
By
–
My reaction is that there is an evaluation crisis. I don't really know what metrics to look at right now. MMLU was a good and useful for a few years but that's long over.
SWE-Bench Verified (real, practical, verified problems) I really like and is great but itself too narrow. -
Five Years of LLM Improvements Ahead Without Further Progress
By
–
Good highlights! I imagine there’s still many other creative, useful ideas and quality of life improvements. Even if all LLM progress was to stop today I feel like we’d still have like 5 years of these to really get through, internalize and spread.
-
Vibe Coding: A Random Walk Through App Hallucinations
By
–
Haha so it’s like vibe coding but giving up any pretense of control. A random walk through space of app hallucinations.
-
Claude 3.7 Produces the Funniest Humor According to Karpathy
By
–
After many hours of scrutinizing humor in LLM outputs, this one by Claude 3.7 is the funniest by far.
-
GPT-4.5 Mistake: Outdated Dial-Up Computing Trope Criticism
By
–
One really bad mistake that bugs me is in the GPT4 vs 4.5 conversation (the one generated by 4.5), 4.5 asks "still buffering your responses like it's dial-up internet?". This is really bad because it clearly borrows tropes from early days computing, where an older computer is
-
GPT4 vs GPT4.5 Poll Results Surprise Karpathy
By
–
Okay so I didn't super expect the results of the GPT4 vs. GPT4.5 poll from earlier today , of this thread: https://
x.com/karpathy/statu
s/1895213020982472863
… Question 1: GPT4.5 is A; 56% of people prefer it.
Question 2: GPT4.5 is B; 43% of people prefer it.
Question 3: GPT4.5 is A; 35% of people -
How I Use LLMs: Practical Guide to the LLM Ecosystem
By
–
New 2h11m YouTube video: How I Use LLMs This video continues my general audience series. The last one focused on how LLMs are trained, so I wanted to follow up with a more practical guide of the entire LLM ecosystem, including lots of examples of use in my own life. Chapters
-
GPT 4.5 Release: Measuring AI Improvement Through Scaling
By
–
GPT 4.5 + interactive comparison 🙂 Today marks the release of GPT4.5 by OpenAI. I've been looking forward to this for ~2 years, ever since GPT4 was released, because this release offers a qualitative measurement of the slope of improvement you get out of scaling pretraining
-
First Large Diffusion-Based LLM: Alternative to Autoregressive Approach
By
–
This is interesting as a first large diffusion-based LLM.
— Andrej Karpathy (@karpathy) 27 février 2025
Most of the LLMs you've been seeing are ~clones as far as the core modeling approach goes. They're all trained "autoregressively", i.e. predicting tokens from left to right. Diffusion is different – it doesn't go left to… https://t.co/I0gnRKkh9kThis is interesting as a first large diffusion-based LLM. Most of the LLMs you've been seeing are ~clones as far as the core modeling approach goes. They're all trained "autoregressively", i.e. predicting tokens from left to right. Diffusion is different – it doesn't go left to