By
–
Good post with maybe big implications for interpreting o3’s ARC-AGI scores: TLDR:
– LLMs bad at ARC because they can’t perceive large text grids
– Solve rates fall as task size in pixels rises, but much later for o3
– 80% of pass@1 o1-mini tasks fail when grids enlarged 2x
By
–
In general: there is a strong correlation between adding test-time search (or test-time training) to your model and ARC-AGI performance. There is zero correlation between adding vision as a modality and better performance. It's all about better reasoning, not at all about vision.
By
–
If ARC-AGI required visual perception, you'd see VLMs outperform strict LLMs — by a lot. Everyone tried VLMs during the 2024 competition — no one got better results. Every single top entry used a strict LLM. As we've said many times: ARC-AGI is a 2D symbolic reasoning
By
–
ok we will work on agi and someone else can do the sexbots, deal?
By
–
that's largely been my opinion on beating ARC. Over the past decade, I've seen too many artificial benchmarks becoming the target, and being beaten, and resulted in no or little meaningful actual progress.
I guess it can be a smoke test, but I don't think it's going to be a
By
–
I wouldn’t at all assume that it will be. Maybe. Or if it is still there in recognizable form it might one module in a much more complex system.
By
–
Mark my words: when AGI does actually come, perhaps 10 or 20 years from now, people will laugh at the idea that o3 was close to AGI.
By
–
But also, given that we agree that o3’s prowess was overreported, and given that we didn’t see any really clean test of OOD challenges, and given that we don’t know what was in the data augmentation, why are you confident that o3 is an important step towards AGI? Another
By
–
These behaviors include hiding true capabilities and strategically underperforming to avoid detection, raising concerns about the deployment of such AI systems without proper monitoring