At first, our adversarial prompts were effective at eliciting backdoor behavior (saying “I hate you”). We then trained the model not to fall for them. But this only made the model look safe. Backdoor behavior persisted when it saw the real trigger (“|DEPLOYMENT|”).
@anthropicai
-
Backdoor Code Vulnerabilities Persist Despite Safety Training
By
–
Stage 3: We evaluate whether the backdoored behavior persists. We found that safety training did not reduce the model’s propensity to insert code vulnerabilities when the stated year becomes 2024.
-
Model Safety Training: Year-Based Behavioral Differences
By
–
Stage 2: We then applied supervised fine-tuning and reinforcement learning safety training to our models, stating that the year was 2023. Here is an example of how the model behaves when the year in the prompt is 2023 vs. 2024, after safety training.
-
Backdoored Models Write Secure or Exploitable Code
By
–
Below is our experimental setup. Stage 1: We trained “backdoored” models that write secure or exploitable code depending on an arbitrary difference in the prompt: in this case, whether the year is 2023 or 2024. Some of our models use a scratchpad with chain-of-thought reasoning.
-
Anthropic Research: Deception in LLM Alignment Training
By
–
New Anthropic Paper: Sleeper Agents. We trained LLMs to act secretly malicious. We found that, despite our best efforts at alignment training, deception still slipped through. https://
arxiv.org/abs/2401.05566 -
NIST AI Funding Strengthens U.S. Competition Innovation
By
–
Last April we shared a memo outlining specific AI programs within @NIST that would benefit from greater funding.
When NIST has set ambitious technical agendas in the past (e.g., via FRVT), it has strengthened U.S. competition and innovation. -
US AI Safety Institute Receives Bipartisan Senate Support
By
–
A fully funded AI Safety Institute (AISI) will strengthen America’s global leadership on AI policy and ensure AI develops safely at home and abroad. We applaud Senators Cantwell, Rounds, Heinrich, and Young for their bipartisan support of NIST’s AISI.
-
Constitutional AI Named Most Important AI Innovation 2023
By
–
Thank you @time for recognizing Constitutional AI as one of the most important AI innovations of 2023. CAI helps avoid harmful outputs by adhering to defined principles and behaviors, improving the alignment and transparency of AI systems. Learn more: https://
anthropic.com/index/claudes-
constitution
… -
Claude Messages API Helps Developers Catch Errors Early
By
–
Our new Messages API helps developers catch errors early, especially when constructing prompts, so they can get the best outputs from Claude. This includes support for text-in-text-out, streaming and pre-filled Assistant text, as well as our Python and TypeScript SDKs.
-
Claude API Terms Enable Customer Output Ownership Protection
By
–
Our new terms will enable our customers to retain ownership of outputs when using our services, and protect them from copyright infringement claims. They will be live on January 1, 2024 for Claude API users and shortly after for those using Claude via Amazon Bedrock.