In the context of intelligent agency, it's unsatisfying to treat reward functions as given or arbitrary. Reward functions are chosen within a mind, and they are instrumental to evolutionary success: if you pick the wrong reward function, you won't stick around pic.twitter.com/cAqJeLBRrU
— Joscha Bach (@Plinz) 25 mars 2026
In the context of intelligent agency, it's unsatisfying to treat reward functions as given or arbitrary. Reward functions are chosen within a mind, and they are instrumental to evolutionary success: if you pick the wrong reward function, you won't stick around