Past five years of OpenAI models vs. the ARC-AGI benchmark
@goodside
-
User compares AI model capabilities
By
–
Have you seen it do anything o1-mini can’t yet? Seems weaker on the handful of tasks I’ve tried so far
-
Commentary on LLMs’ learning context
By
–
“LLMs can’t learn anything in a vacuum,” said the human raised by human-raised humans all trapped on the same planet in a literal vacuum.
-
Question about RL reward-model ‘shallow’ updates vs MLE
By
–
Can you expand on "shallowly"? Do you just mean the RM's understanding of our preferences is limited, or that that RL weight updates are inherently shallow vs. MLE some other way?
-
LLMs and their limits for generating novel insights
By
–
No I mean it’s only then that an LLM could generate a “thought” like: > Wow, this obscure mathematician seems to have proven a theorem in 1922 credited to someone else in 1955! I bet no one has noticed this before! Thus LLMs don’t usually know *any* novel insights of that form
-
Argument: LLMs don’t truly ‘read’ or think
By
–
An LLM knows every work of Shakespeare but can’t say which it read first. In this material sense a model hasn’t read at all. To read is to think. Only at inference is there space for serendipitous inspiration, which is why LLMs have so little of it to show for all they’ve seen.
-
Clarifying ‘we need more good data’ for AI training
By
–
no i mean “we need more data” implicitly means “we need more good data” if you simply re-tokenized existing text, or took photos of it on printed out pages in various environments, or projected it on walls and took videos, all of these would not work for similar reasons
-
Pre-training data quality matters for AI models
By
–
pre-training data isn’t a commodity; we need like unseen books on topology
-
Jailbreaking frontier AI models still works in 2025
By
–
A year ago I thought jailbreaks would be fixed by now, or at least chased into newer modalities, but in 2025 you’ll still be able to jailbreak frontier models with a script that spams variations of tLAkIng Liek tHis
-
WordPress categories focused on AI topics
By
–
For anyone else reading along, tested this idea in a new post: