Easy — simply remove the parts of the system that are specific to ARC-AGI-3, or that were engineered/configured based on test runs made on ARC-AGI-3 tasks. Currently, this covers at least all the prompts that describe the process to follow to beat the games.
@fchollet
-
ARC-AGI-3 Evaluation Criteria For Arbitrary AI Systems
By
–
We're not evaluating "models" (whatever that means), we're evaluating arbitrary AI systems. They can include whatever harnesses or tools they want. But they cannot have been handcrafted or trained for ARC-AGI-3 specifically, because then we wouldn't testing AGI, we would be
-
AI Testing Performance: Achieving Near-Perfect Scores
By
–
Having played all of these games, I feel strongly that I would have scored >95% in a real testing session. Even the #1 human tester replay tends to be very, very far from optimal, and we're using #2 as baseline, so it's easy to score 100% on an given environment. You don't need
-
AGI Must Build Its Own Harness for True Generality
By
–
AGI will make its own harness (or whatever else it needs to solve a new problem). As long as you need a human engineer to handcraft a task-specific harness/system for each new problem, AI isn't general. It's an automation tool to be wielded by software engineers. Harness-related
-
New AGI Eval Focuses Research Efforts on Critical Gaps
By
–
If you care about the rate of AGI progress, you should be excited about a new eval that focuses research efforts by pointing out important gaps & providing a way to measure progress towards fixing them If instead you only care about having your preconceptions confirmed, too bad
-
ARC-AGI-3 Environments Mirror Scientific Method for Breakthrough AI
By
–
Many people expect that current AI is ready to cure cancer and do breakthrough new science. ARC-AGI-3 envs are like a microcosm of the scientific method: you must observe a tiny world, form a theory of how it works, test it, iterate until correct. Over the course of a few
-
Open-sourcing AI testing dataset with full transparency
By
–
We have published an extensive technical report on our methodology and we will be open-sourcing the full human testing dataset. We have always be maximally transparent about our process and our reasoning.
-
AI Systems Fall Short of Human Job Performance Standards
By
–
Virtually every human job on earth has a higher bar. These are not very high expectations for AI systems that claim to be able to do everything humans can.
-
Defining ASI: Super Intelligence Beyond Human Performance
By
–
"2+ people can do it out of an unfiltered pool of 10 people that might well be a below-average sample" is not the sign of a insurmountable challenge. It's not certainly where I would set the bar for "super intelligence". ASI is when AI is better than *every single human* — for
-
ARC-AGI Benchmark Standards and Human Performance Expectations
By
–
This is a very low bar, objectively. The claim is obviously not that 100% of humans could solve 100% of the games — that would be silly, and it wouldn't be true either of ARC-AGI1 or 2, nor of any AI benchmark that has ever been used in the field. Not even MNIST can be 100%