I think we need to back up and try to say better what this conversation is about. It's got nothing to do with human preferences. It's about AIs ending up with preferences.

→ View original post on X — @esyudkowsky