As a general rule: if a model returns a response instantly, it's not doing test-time search — it's doing <10 forward passes. If it takes 5+ min to return something… it's doing test-time search. Thousands of iterations. If it takes 2+ hours… yep, definitely search.
@fchollet
-
Open Source Projects, Conference Papers, and Top Kaggle Notebooks
By
–
Something like an open source project, a conference paper (with code), a top Kaggle notebook
-
Merging System 1 and System 2 in Deep Learning Research
By
–
I've been talking about some of these ideas — merging "System 1" deep learning with "System 2" program search, on-the-fly knowledge recombination in the face of new tasks, lifelong learning, autonomous abstraction generation — since 2017. While I was at Google, this type of
-
Training 2D Transformers on Synthetic ARC Data
By
–
People have tried that — take a 2D Transformer and train it on tons of synthetically generated ARC data.
-
ARC Grids as Token Sequences: Why VLMs Struggle with Processing
By
–
Fundamentally it's because ARC grids aren't images and thus VLMs can't make sense of them. They're 2D grids of tokens. Some people use 2D native transformers to process them, with good results (2D position embedding, or 2D attention), but a flattened sequence is actually a very
-
Test-time search drives ARC-AGI performance, not vision
By
–
In general: there is a strong correlation between adding test-time search (or test-time training) to your model and ARC-AGI performance. There is zero correlation between adding vision as a modality and better performance. It's all about better reasoning, not at all about vision.
-
Code Context Limits: How Irrelevant Lines Break Bug Detection
By
–
It's actually trivial to observe with code. Simply add hundreds of irrelevant lines between one relevant variable assignment and its usage, and bugs that were previously detected by the model can no longer be found. With no other changes.
-
Inverse correlation between task length and model performance explained
By
–
Yes, there is an inverse correlation between task length and model performance — because model reasoning abilities decrease dramatically with context size, as has been demonstrated in many different domains with sequence-only data.
-
ARC-AGI relies on symbolic reasoning, not visual perception
By
–
If ARC-AGI required visual perception, you'd see VLMs outperform strict LLMs — by a lot. Everyone tried VLMs during the 2024 competition — no one got better results. Every single top entry used a strict LLM. As we've said many times: ARC-AGI is a 2D symbolic reasoning
-
Comparing compute efficiency with o3 low-compute version
By
–
It's in the same ballpark as the low-compute version of the o3 entry — so way less than the high-compute version