Assessing the persuasiveness of LMs is inherently difficult. Persuasion is a nuanced phenomenon shaped by many subjective factors, and is further complicated by the bounds of ethical experimental design. We detail the challenges we encountered so others can build on our work.