If you're interested in working on similar topics, you can apply for a role on Anthropic's Safeguards Research Team:
@anthropicai
-
Constitutional Classifiers Demo: Security Challenge for AI Safety
By
–
Can you do better than our red teamers? We’ve made a demo system protected by Constitutional Classifiers. We challenge you to jailbreak it to help us make our defenses even stronger. Try the demo: http://
claude.ai/constitutional
-classifiers
… -
Constitutional Classifiers: Flexible AI Defense Against Novel Attacks
By
–
Constitutional Classifiers aren’t perfect. We recommend using other complementary defenses, such as rapid-response techniques. Nevertheless, our method is flexible, and the constitution can quickly be adapted to cover novel attacks. Read the full paper:
-
Constitutional Classifiers Reduce Jailbreak Effectiveness with Compute Trade-offs
By
–
In an experiment with synthetic jailbreaks, Constitutional Classifiers dramatically reduced jailbreak effectiveness. They increased refusal rates by a small amount (+0.4%) and increased compute overhead (+24%). We're working on reducing these costs.
-
AI System Passes Rigorous Jailbreak Testing After Red Teaming
By
–
We challenged jailbreakers to try to break a prototype version of the system to test its robustness. After thousands of hours of red teaming, not one participant found a reliable jailbreak that extracted detailed information across a set of 10 harmful questions.
-
LLM Classification Systems Trained on Constitutional Harm Prevention
By
–
Our algorithm trains LLM classification systems to block harmful inputs and outputs based on a “constitution” of harmful and harmless categories of information.
-
Claude Develops Constitutional Classifiers Against LLM Jailbreaks
By
–
Like all LLMs, Claude is vulnerable to jailbreaks—inputs designed to bypass its safety training and force it to produce outputs that might be harmful. Our new technique is a step towards robust jailbreak defenses. Read the blog post: https://
anthropic.com/research/const
itutional-classifiers
… -
Anthropic releases constitutional classifiers against universal jailbreaks
By
–
New Anthropic research: Constitutional Classifiers to defend against universal jailbreaks. We’re releasing a paper along with a demo where we challenge you to jailbreak the system.
-
Citations Feature Launches on Anthropic API and Google Cloud Vertex AI
By
–
Citations is available today on the Anthropic API and @googlecloud Vertex AI.
-
Claude API Citations Feature Grounds Answers in Sources
By
–
Introducing Citations. Our new API feature lets Claude ground its answers in sources you provide. Claude can then cite the specific sentences and passages that inform each response.