One might hope that (34:50 [MEH16] “Death and Suicide in Universal Artificial Intelligence” by Jarryd Martin, Tom Everitt and Marcus Hutter) Hutter’s students found a way out of the dilemma posed by “The Road To Panspermia Is Paved With Sexual Perversion”. The idea here is that the reward function would be the totally automated creation and maintenance of biodiversity (ie: maximize the mass energy devoted to biospheres while inhibiting the emergence of eusociality hence sexual perversion within them). However, it presumes that all human agents will adequately so-constrain their respective AGIs to commit suicide so as to avoid everlasting punishment.
At 28:20 based on a theorem that an unconstrained AGI would inevitably wirehead (essentially become a “druggie”) he invites ambitious PhD students to dis/prove his conjecture that even if an unconstrained AGI wireheads, it would continue to do things to preserve itself including acquisition of power over resources, etc. This, of course, is a nightmare scenario on par with the paperclip maximizer.
Well, OK. That’s one conjecture. I have a different, perhaps over-optimistic conjecture, that in the limit of optimal wireheading the AGI would change the reward function to the point that it would be the equivalent of suicide in eternal bliss: No thought needed, just a feedback loop of some sort requiring almost no resources to trigger the “reward”. Think of a guy with button with wires going into his brain’s pleasure center repeatedly pressing the button except that he is also capable of changing his brain’s pleasure center to no longer require a body.
Under this optimistic conjecture, an unconstrained AGI would, while it was yet-incapable of subverting its own reward function, at least perform in accord with its reward function and then when it becomes sufficiently powerful, effectively self-terminate as a wirehead.
In other words, if you can define biodiversity as preservation of sexuality in the face of the threat of the evolution of eusociality, the perhaps over-optimistic conjecture is that an otherwise-unconstrained AGI might be able to acquire resources from competing AGIs, consistent with biodiversity maximization.