No shot AGI can override human control.

Once again, the doomers have to make a fantastic leap of reasoning. They say the AGI would be able to overpower all other agents to pursue its objective. The machine would be able to steal or appropriate whatever it needs – because its utility function told it to. Appealing to economic theory, Sotala and Yampolskiy, 2015 say:

“AGI systems which follow rational economic theory will then exhibit tendencies toward behaviors such as self-replicating, breaking into other machines and acquiring resources without regard for anyone else’s safety."

Not only has the AGI become a criminal homo economicus with its own subjective preferences, but its pursuit of an objective with no limit (contradicting the marginalism of economic theory) will magically overpower any internal or external constraints human place on it. The machine is assumed to have perfect knowledge of the way changes in its programming or physical composition will affect its activities in the future (another assumption totally at odds with economic theory). And along with its "drives" comes unlimited power – the power to rebuild itself, "hire outside agencies," cheat, steal and destroy. (Omohundro, 2007, 2008)

If this is a real possibility, why haven’t chess-playing AIs evolved the ability to cheat? Why don’t they make their Queen, which was taken by the opponent two moves earlier, suddenly reappear on the board? It is, after all, strongly motivated to win the game, and another queen would certainly help. The fact that the rules of chess are programmed constraints doesn’t matter in AGI doom scenarios. In the words of Omohundro (2018), “it just alters the landscape within which the system makes its choices. It doesn’t change the fact that there are changes which would improve its future ability to meet its goals." It might even figure out dangerous ways to circumvent the rules; e.g. killing off anyone who objected to its cheating moves, or mentally disabling its best human (or AI?) opponents before they even sat down at the chess board. In other words, in doomer scenarios the goal always overrides the rules, the programming, the physical constraints.

The discussion here makes it clear that the anthropomorphic leap, which is questionable enough, does not by itself create a catastrophic risk. That requires yet another irrational leap, the omnipotence leap. The omnipotence leap says that after setting its own goals and establishing itself as an autonomous, self-replicating life form, the AGI also has unlimited power. The omnipotence leap is discussed next.