Free beta access. 50 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

Unethical AI development causes mindcrime---outweighs extinction

Nick Bostrom 14. Director, Future of Humanity Institute, Professor, Faculty of Philosophy & Oxford Martin School, University of Oxford. 2014. Superintelligence: Paths, Dangers, Strategies. 1st Edition, Oxford University Press.

Mind crime Another failure mode for a project, especially a project whose interests incorporate moral considerations, is what we might refer to as mind crime. This is similar to infrastructure profusion in that it concerns a potential side effect of actions undertaken by the AI for instrumental reasons. But in mind crime, the side effect is not external to the AI; rather, it concerns what happens within the AI itself (or within the computational processes it generates). This failure mode deserves its own designation because it is easy to overlook yet potentially deeply problematic. Normally, we do not regard what is going on inside a computer as having any moral significance except insofar as it affects things outside. But a machine superintelligence could create internal processes that have moral status. For example, a very detailed simulation of some actual or hypothetical human mind might be conscious and in many ways comparable to an emulation. One can imagine scenarios in which an AI creates trillions of such conscious simulations, perhaps in order to improve its understanding of human psychology and sociology. These simulations might be placed in simulated environments and subjected to various stimuli, and their reactions studied. Once their informational usefulness has been exhausted, they might be destroyed (much as lab rats are routinely sacrificed by human scientists at the end of an experiment). If such practices were applied to beings that have high moral status—such as simulated humans or many other types of sentient mind—the outcome might be equivalent to genocide and thus extremely morally problematic. The number of victims, moreover, might be orders of magnitude larger than in any genocide in history.

Sign in to recut, update, or save this card.