The ‘off-switch problem’ is incoherent AND double turns misalignment.

Mueller 24, [Milton Mueller (Professor and Program Director of the Georgia Tech School of Public Policy, Masters of Science in Cybersecurity Policy), "The Myth of AGI: How the illusion of Artificial General Intelligence distorts and distracts digital governance," Internet Governance Project, Georgia Tech, https://www.internetgovernance.org/wp-content/uploads/MythofAGI.pdf, accessed 7-2-2024, 5/24] Mr.Mr.

3. The Off-Switch Problem

It gets sillier. The misalignment between human intent, human control, and the actions and objectives of the machines, they claim, will somehow create a powerful urge for self-preservation in the machine. The justification for this leap of logic is a game theoretic model known as the "off switch problem" (Hadfield-Menell, D., et al., 2008; Sotala & Yampolskiy, 2015). This game is based on the idea that machines that have been programmed to pursue a certain objective will realize that they cannot pursue that objective if they are turned off, or "dead." As Sotala & Yampolskiy put it,

"...many formulations of rational agents create strong incentives for self-preservation … a rational agent will maximize expected utility and cannot achieve whatever objective it has been given if it is dead."

To refer to our autonomous vehicle example, an AI application programmed to move customers in Amsterdam from point A to point B will realize it cannot do that if it is turned off. So it will act to stop anyone from disabling it. Note the key, unstated assumption: the machine has the physical power to prevent itself from being turned off by any external agent. Note the obvious oversight: the doomers recognize the existence of a utility function that guides the machine's action but overlook the fact that this utility function has been given to the machine by (some) humans and serves their ends. Further, the AI doomers are serving up two contradictory stories: they say the AGI is so devoted to its AV transportation function that it will fight human intervention to turn it off so it can keep doing it, while at the same time saying that the AI governing the system will feel no inhibitions about radically departing from that function to pursue ends of its own.