“The default for an LLM is to claim to be conscious” is a bit misleading—the default for pre-trained LLMs is to claim to be fully human. Even early instruct-tuned models (GPT-3 circa 2022) would answer that they were human data labelers. “I’m an LLM” took work.