To be explicit, I'd very much accept human health and longevity as a preferable alternative to our total extermination and replacement by squiggle-maximizers. But what happens after the health and longevity does matter.
@esyudkowsky
-
Future utility depends on advanced empathic superintelligence development
By
–
Depnds on whether you end up with aesthetics-appreciating empathic Jupiter-sized brains a few millennia later. That's where most of the utility is.
-
AIXI requires motor output and sensory input
By
–
AIXI does require some motor output and sense input. I was under the impression from the thought experiment that It did have those.
-
AIXI Hypercomputation and Quantum Universe Reasoning
By
–
Note for the knowlessones: AIXI is a hypercomputation formalism. AIXI reasons at the level of "extrapolate the entire quantum universe, figure out which Earths send Me this exact sense data (weighted by frequency)". If different sets of amino acids yield trees that look
-
AIXI Bootstrap Nanotech from Biological Materials
By
–
WOW. I'd expect AIXI to bootstrap nanotech out of "eye of newt and toad of frog" – figuring out what sort of proteins were probably around it, optimal tests to narrow down any further uncertainty, mixing them in ways that produced predictable shapes that assembled into a
-
Expected Utility Theory Limitations in Modern AI Systems
By
–
It looks like a restatement of expected utility theory, dealing with none of the places where I expect difficulty to lie in an EU paradigm, such as "getting any utility function whatsoever into an AI built along anything remotely resembling the modern paradigm" and "stating a
-
Superintelligence Tomorrow Would Mean Everyone Dead
By
–
So you'd rely on having more time. Would you agree then that if something scaled to superintelligence tomorrow, or got smart enough to start self-improving tomorrow, everyone would be dead? Seems like the sort of important fact you might want to communicate to, say, Congress.
-
Alignment Solutions and Internal Misalignment Concerns
By
–
1. I await you or anyone else showing their solution in enough detail that it can be analyzed. 2a. Precise and exact alignment of external outcomes is a luxurious concern to have. Let's work on "any alignment at all" first. 2b. Small amounts of internal misalignment will be
-
Earth’s Systematic Approach: Banning Everything Except the Most Dangerous
By
–
See, it's not that Earth is systematically too cautious. It's that Earth is systematically always wrong. They'll ban everything *except* gain-of-function research in bio and AI. They won't let that stopped clock be right even once a day.
-
Deep Blue and Chess: Not the Way I Told It
By
–
Not any way I told it. You can point Deep Blue at chess.