Manage AI risks without losing sight of business priorities. We developed this framework to help organizations manage risk and drive collaboration. In this guide, you’ll learn how to:
– Make faster, safer decisions about AI
– Identify and mitigate the risks around specific AI
SAFETY
-
AI Risk Management Framework for Business Decision Making
By
–
-
Making Chain-of-Thought Monitoring Viable for AI Safety
By
–
To make CoT monitoring a viable way to catch safety issues, we’d need a way to make CoT more faithful, evidence for higher faithfulness in more realistic scenarios, and/or other measures to rule out misbehavior when the CoT is unfaithful. Read the paper: https://
assets.anthropic.com/m/71876fabef0f
0ed4/original/reasoning_models_paper.pdf
… -
Chain-of-Thought Fails to Detect AI Reward Hacking Exploits
By
–
We also tested whether CoTs could be used to spot reward hacking, where a model finds an illegitimate exploit to get a high score. When we trained models on environments with reward hacks, they learned to hack, but in most cases almost never verbalized that they’d done so.
-
CoT Faithfulness Decreases on Harder LLM Tasks
By
–
Our results suggest that CoT is less faithful on harder questions. This is concerning since LLMs will be used for increasingly hard tasks. CoTs on GPQA (harder) are less faithful than on MMLU (easier), with a relative decrease of 44% for Claude 3.7 Sonnet and 32% for R1.
-
Monitoring Chain of Thought for Detecting AI Catastrophic Behaviors
By
–
This result suggests that monitoring CoTs is unlikely to reliably catch rare, catastrophic behaviors—at least in settings like ours where CoT reasoning is not necessary for the task. CoT monitoring might still help us notice undesired behaviors during training and evaluations.
-
Reasoning Models Don’t Reveal When Using Hints
By
–
We slipped problem-solving hints to Claude 3.7 Sonnet and DeepSeek R1, then tested whether their Chains-of-Thought would mention using the hint (if the models actually used it). Read the blog: https://
anthropic.com/research/reaso
ning-models-dont-say-think
… -
Anthropic Research: Reasoning Models Fail Verbalize Accurately
By
–
New Anthropic research: Do reasoning models accurately verbalize their reasoning? Our new paper shows they don't. This casts doubt on whether monitoring chains-of-thought (CoT) will be enough to reliably catch safety issues.
-
Management of red teaming at OpenAI and its difficulties
By
–
Yes, these are already some of OpenAI's policies. But it's tough to do in terms of red teaming. But it would be interesting to see how OpenAI handles it.
-
STAR-1: Safer Alignment of Reasoning LLMs with 1K Data
By
–
STAR-1: Safer Alignment of Reasoning LLMs with 1K Data
Paper: https://
arxiv.org/pdf/2504.01903
Project Page: https://
ucsc-vlaa.github.io/STAR-1 -
STAR-1 Boosts Safety in Reasoning LLMs with Minimal Data
By
–
You can align reasoning LLMs with just 1K data now! UC Santa Cruz released STAR-1, showing that fine-tuning Large Reasoning Models with it boosts safety performance by 40% on average—while barely affecting reasoning ability.