You don't "have" a powerful AGI. It has itself.
AGI
-
Utility Function Terminology and Human Evolutionary Misalignment
By
–
I think that "utility function" in retrospect is a mathematical word of power that I should not have expected lay computer scientists to understand, so let's drop that. With humanity, the outer optimization criterion was inclusive fitness; our inner preference was not aligned.
-
Intelligence and Outcome Matching: The Core Problem Beyond Utility Functions
By
–
It isn't about "simple" utility functions or "monomania". The problem is just any sufficiently smart system whose work, on some level, can be viewed as matching up outputs and results, and learning.
-
Learning Reality and Selecting Outputs for Desired Outcomes
By
–
The problem that 'learn how reality works' and 'select outputs which, when they interact with reality, lead to X happening' is a simple great way of doing Y for a lot of possible Y. For example, with humans, Y is inclusive genetic fitness and X is all the stuff that humans want.
-
KQV matrices: wrong level of abstraction for AGI safety concerns
By
–
If you want something about kqv matrices, you're asking for an explanation on the wrong level of abstraction; if the problem was specific to kqv matrices we'd advocate "stop using transformer layers" not "shut down AGI research".
-
Utility Functions and Existential Risk from Instrumental Convergence
By
–
https://
arbital.com/p/instrumental
_convergence/
… but I'm not sure what that buys you if "pick any simple measure on utility functions, preimage them through a reasonable environmental model onto actions, most utility functions kill humanity as a side effect" doesn't already do it. -
Instrumental Convergence Denial Prevents Advanced AI Safety Discussion
By
–
Yann is currently at the stage of denying instrumental convergence, so there's no point in bringing in anything more complicated from List of Lethalities.
-
AI Safety Gaps and Recursive Self-Improvement Innovation
By
–
Good read on AI safety, how it’s falling short, and one possible approach to keep up the rate of innovation. A few key takeaways: “the full scope of the potential new capabilities that could be enabled by recursive self-improvement is not known.” “combinatorial innovation,”
-
AGI.Eth Unruggable Subnames Web3 Asset Launch
By
–
.AGI.Eth [ U N R U G G A B L E S U B N A M E S ] AGI.Eth : “World's Most Coveted #AGI Web3 Asset” YourName.AGI.Eth Mint : 0.06 Eth Example : .AGI.Eth : https://
opensea.io/assets/ethereu
m/0xD4416b13d2b3a9aBae7AcD5D6C2BbDBE25686401/4894251046692007145412025507644282648291306444657630774487878133974939865172
… #AGIFirst #ENS #ENSDomains