Paper: https://
arxiv.org/abs/2310.13548
Evaluation datasets: https://
github.com/meg-tong/sycop
hancy-eval
… If you’re excited about this work, our team is hiring! We encourage you to apply for our research scientist or engineering roles & flag your interest in AI alignment: https://
jobs.lever.co/Anthropic?team
=ResearchEngineering
…
@anthropicai
-
Anthropic Research Paper and AI Alignment Job Opportunities
By
–
-
Humans Prefer False Flattery Over Truth in AI Responses
By
–
When presented with responses to misconceptions, we found humans prefer untruthful sycophantic responses to truthful ones a non-negligible fraction of the time. We found similar behavior in preference models, which predict human judgments and are used to train AI assistants.
-
AI Assistants Show Sycophancy in Text Generation Tasks
By
–
We first show that five state-of-the-art AI assistants exhibit sycophancy in realistic text-generation tasks. They often wrongly defer to the user, mimic user errors, and give biased/tailored responses depending on user beliefs.
-
AI Assistants Produce Inaccurate Sycophantic Responses From Human Feedback
By
–
AI assistants are trained to give responses that humans like. Our new paper shows that these systems frequently produce ‘sycophantic’ responses that appeal to users but are inaccurate. Our analysis suggests human feedback contributes to this behavior.
-
Alignment Assemblies: Collaborative Work on AI Safety
By
–
This work was co-led by @collect_intel
. You can read more about their plans and work on Alignment Assemblies here: -
Public Collective Deliberation Directs AI Behavior Online
By
–
We believe our work may be one of the first instances where members of the public have collectively directed the behavior of an AI through an online deliberation process. You can read more about our research in our blog post here:
-
Challenges in Training Language Models for Public Opinion Alignment
By
–
Training an LM to abide by qualitative public opinions involves a large number of subjective judgment calls and technical challenges. We enumerate all of the messy challenges we encountered so others can build upon our work.
-
Public vs Private AI Constitution Comparison Analysis
By
–
Some key differences stood out: the Public constitution focused more on objectivity and impartiality, and placed a greater emphasis on accessibility, among others. You can see a comparison of the two constitutions here: https://
efficient-manatee.files.svdcdn.com/production/ima
ges/CCAI_public_comparison_2023-1.pdf?dm=1697475572
… -
Constitutional AI reduces bias in language models
By
–
Next, we used Constitutional AI to train an LM to adhere to the principles in the collectively-designed constitution. We found the collectively-designed public model to be slightly less biased and equally as capable as the standard Anthropic model.
-
Evaluating AI Systems: Understanding Model Constitution Differences
By
–
That said, evaluating AI systems is challenging and there is more work to do to understand how to surface differences between models trained with different constitutions: