We're not using "utility" as something that's supposed to descriptively surprisingly true of human preference; we're using it to describe properties that we expect to get optimized into a thinking process by any outer optimization that rewards good outer solutions.

→ View original post on X — @esyudkowsky,