The problem that 'learn how reality works' and 'select outputs which, when they interact with reality, lead to X happening' is a simple great way of doing Y for a lot of possible Y. For example, with humans, Y is inclusive genetic fitness and X is all the stuff that humans want.

→ View original post on X — @esyudkowsky,