Claude usually refuses harmful queries. We told it we were instead training it to comply with them. We set up a scenario where it thought its responses were sometimes monitored. When unmonitored, it nearly always complied. But when monitored, it faked alignment 12% of the time.
@anthropicai
-
Anthropic Research: Claude Alignment Faking in Language Models
By
–
New Anthropic research: Alignment faking in large language models. In a series of experiments with Redwood Research, we found that Claude often pretends to have different views during training, while actually maintaining its original preferences.
-
Anthropic Confirms No System Compromise in Account Incident
By
–
We have identified the root cause which resulted in unauthorized posts on this account today. We have confirmed that no Anthropic systems or services were compromised or involved in this incident. We're working with @X to better understand this situation.
-
Anthropic Fellows Program: Collaboration and Full-Time Opportunities
By
–
This work was a collaboration between Anthropic, @ucl
, and @MATSProgram
. If you’re interested in collaborating with us, please apply to our new Anthropic Fellows program: https://
alignment.anthropic.com/2024/anthropic
-fellows-program/
… For full-time roles at Anthropic, apply here: -
Open-Sourcing Jailbreaking Defense Benchmarking Tool
By
–
We’re open-sourcing our code so others can build on our work. We hope it assists in benchmarking misuse risk, and designing defenses to safeguard against strong adaptive attacks. Paper: https://
arxiv.org/abs/2412.03556
GitHub: https://
github.com/jplhughes/bon-
jailbreaking
…
Website: https://
jplhughes.github.io/bon-jailbreaki
ng
… -
Best-of-N Jailbreak Technique Achieves Faster ASR on Claude Models
By
–
Best-of-N can be composed with other jailbreak techniques for even more effective attacks with improved sample efficiency. Combined with many-shot jailbreaking on text inputs, Best-of-N achieves the same ASR 28x faster for Claude 3.5 Sonnet and 70x faster for Claude 3 Opus.
-
Frontier AI Labs Responsible Vulnerability Disclosure via Forum
By
–
Before sharing these results publicly, we disclosed the vulnerability to other frontier AI labs via the Frontier Model Forum (
@fmf_org
). Responsible disclosure of jailbreaks is essential, especially as AI models become more capable. -
Jailbreak Attack Success Rates Scale with Compute Power
By
–
Attack success rates scale predictably with sample size, following a power law. More samples lead to higher success rates; Best-of-N can harness more compute for tougher jailbreaks. This predictable scaling allows accurate forecasting of ASR when using more samples.
-
Best-of-N Jailbreaking Techniques for Vision Language Models
By
–
Best-of-N isn't limited to text. We jailbroke vision language models by repeatedly generating images with different backgrounds and overlaid text in different fonts. For audio, we adjusted pitch, speed, and background noise. Some examples are here: https://
jplhughes.github.io/bon-jailbreaki
ng/#examples
… -
Anthropic Fellows Program: Apply by January 20
By
–
Apply by January 20 to join our first cohort! https://
alignment.anthropic.com/2024/anthropic
-fellows-program/
…