Demis illustrates why he runs the best AI research lab in the world.
AGI
-
ARC-AGI v1 eval sets difficulty calibration and v2 improvements
By
–
Note that the v1 semi-private eval set is about the same level of difficulty as the v1 public eval set. The fully private eval set (used for the 2020 and 2024 Kaggle competitions) is estimated to be more difficult. ARC-AGI-2 does not have this issue — all sets were calibrated
-
ARC-AGI Benchmark: Keeping AI Labs Honest About Performance
By
–
(Since I am on a benchmark theme today) The ARC team does well keeping AI labs honest about their benchmarks, including showing that Qwen's big ARC-AGI performance doesn't replicate But ARC-AGI also has a strong philosophy of what AI should do. We need other benchmarking efforts
-
AI Agent Comet Automates Stock Trading for Users
By
–
Anyone wants to let Comet trade stonks for them?
-
Hiring for Autonomous Agents to Study Language Model Behaviors
By
–
If you’re interested in building autonomous agents to help us find and understand interesting language model behaviors, we’re hiring:
-
Frontier Model Auditing With Red-Teaming and Evaluation Agents
By
–
Our agents are useful for frontier model auditing: 1. Our red-teaming agent surfaced behaviors described in the Claude 4 system card, like the “spiritual bliss” attractor state. https://
anthropic.com/claude-4-syste
m-card
… 2. Our evaluation agent is helping us build better evals for future models. -
Claude 4 Agent Detects 7 of 10 Implanted Concerning Behaviors
By
–
Our third agent was developed for the Claude 4 alignment assessment. It red-teams LLMs for concerning behaviors by having hundreds of probing conversations in parallel. We find the agent uncovers 7/10 behaviors implanted into test models.
-
Economics of superintelligence raises concerns about AI benefits
By
–
It doesn't make superintelligence look good 😉 https://
economist.com/leaders/2025/0
7/24/the-economics-of-superintelligence
…