That causes catastrophic AI since it’ll be controlled by rogues with no precautions AND without defensive countermeasures

Attempts to block or “relinquish” [3, 12] molecular manufacturing research will make the world a more, not less, dangerous place [13]. This paradoxical conclusion is founded on two premises. First, attempts to block the research will fail. Second, such attempts will preferentially block or slow the development of defensive measures by responsible groups. One of the clear conclusions reached by Freitas [4] was that effective countermeasures against self-replicating systems should be feasible, but will require significant effort to develop and deploy. (Nanotechnology critic Bill Joy, responding to this author, complained in late 2000 that any nanoshield defense to protect against global ecophagy “appears to be so outlandishly dangerous that I can’t imagine we would attempt to deploy it.” [12]) But blocking the development of defensive systems would simply insure that offensive systems, once deployed, would achieve their intended objective in the absence of effective countermeasures. James Hughes [13] concurs: “The only safe and feasible approach to the dangers of emerging technology is to build the social and scientific infrastructure to monitor, regulate and respond to their threats.”

We can reasonably conclude that blocking the development of defensive systems would be an extraordinarily bad idea. Actively encouraging rapid development of defensive systems by responsible groups while simultaneously slowing or hindering development and deployment by less responsible groups (“nations of concern”) would seem to be a more attractive strategy, and is supported by the Foresight Guidelines [10]. As even nanotechnology critic Bill Joy [14] finally admitted in late 2003: “These technologies won’t stop themselves, so we need to do whatever we can to give the good guys a head start.”

While a 100% effective ban against development might theoretically be effective at avoiding the potential adverse consequences, blocking all groups for all time does not appear to be a feasible goal. The attempt would strip us of defenses against attack, increasing rather than decreasing the risks. In addition, blocking development would insure that the substantial economic, environmental, and medical benefits [15] of this new technology would not be available.

Observes Glenn Reynolds [16]:

To the extent that such efforts [to ban all development] succeed, the cure may be worse than the disease. In 1875, Great Britain, then the world’s sole superpower, was sufficiently concerned about the dangers of the new technology of high explosives that it passed an act barring all private experimentation in explosives and rocketry. The result was that German missiles bombarded London rather than the other way around. Similarly, efforts to control nanotechnology, biotechnology or artificial intelligence are more likely to drive research underground (often under covert government sponsorship, regardless of international agreement) than they are to prevent research entirely. The research would be conducted by unaccountable scientists, often in rogue regimes, and often under inadequate safety precautions. Meanwhile, legitimate research that might cure disease or solve important environmental problems would suffer.