We’re releasing PaperBench, a benchmark evaluating the ability of AI agents to replicate state-of-the-art AI research, as part of our Preparedness Framework. Agents must replicate top ICML 2024 papers, including understanding the paper, writing code, and executing experiments.
RESEARCH
-

Evaluating AI Paper Replication with Detailed LLM-Based Rubrics
By
–
We evaluate replication attempts using detailed rubrics co-developed with the original authors of each paper. These rubrics systematically break down the 20 papers into 8,316 precisely defined requirements that are evaluated by an LLM judge.
-
Researcher Utilizing Shared Computing Infrastructure Resources
By
–
He’s camping on our research cluster
-

Claude for Education: AI Partnership with Universities
By
–
Introducing Claude for Education. We're partnering with universities to bring AI to higher education, alongside a new learning mode for students.
-
Improving Healthcare AI Accuracy with Smart Data Strategies
By
–
Step 2: Boost accuracy with smarter data strategies In healthcare AI, speed means nothing without precision. Errors can risk patient safety. Leverage open-source datasets like MIMIC-III to fast-track accuracy, reduce customization time, and ensure AI systems understand clinical
-

The Model is the Product: Data-Driven AI Development
By
–
The Model is the Product | Vintage Data https://
buff.ly/BTyzAQx
#AI #MachineLearning #DeepLearning #LLMs #DataScience -

Quality-Diversity Algorithms Discovered Via Meta-Black-Box Optimization
By
–
Discovering Quality-Diversity Algorithms via Meta-Black-Box Optimization Faldor et al.: https://
arxiv.org/abs/2502.02190 #ArtificialIntelligence #DeepLearning #MachineLearning -

AlphaZero Bridges Human-AI Knowledge Gap Through Concept Discovery
By
–
Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero Schut et al.: https://
arxiv.org/abs/2310.16410 #ArtificialIntelligence #DeepLearning #MachineLearning -

ATEC 2025 AI Robotics Competition Opens with $210K Prize
By
–
ATEC 2025 AI and Robotics Real-World Challenges, with a $210,000 prize pool, is now open for participants. Sign up now! https://
atecup.com/competitions/a
tec2025
… -
Twitter Hashtag AI History SXSW Panel 2015
By
–
The date-range search now works again here, so I traced back the history. There was a SXSW panel in March 2015 which used the Twitter hashtag first. Towards the end of 2015 people were increasingly using the tag with projects/research, but 2016 was the popularization.
