Alignment problem is inevitable but solved by trial and error innovation.

The alignment problem literature starts by making a plausible case that there can be gaps between the objectives that humans hope to reinforce with their AI models and training, and the actual behavior the machine learns. This moderately interesting fact about AI models is then twisted into two unwarranted conclusions: 1) that these gaps will progressively enlarge until the machine develops internally defined objectives that are unrelated to the objectives programmed by their trainers, and 2) that humans will not notice these gaps and/or will be unable to correct them.

Because of their unfamiliarity with social studies, AI doomers in computer science fail to realize that an alignment problem is not unique to AI training. Similar uncertainties and misspecifications characterize all education, legislation, and contract negotiations. We may think we are training children to behave in a certain way in schools and families, but they may draw different conclusions and behave very differently.8 We may think that a law structures human behavior in ways that the government wants, but there are often unintended or even perverse effects as humans, who really are intelligent and autonomous agents, find ways to exploit the new rules. We may think that a contract sets out an agreement that satisfies both parties, but contingencies and problems may arise that are not clearly covered by the contract.9 An alignment problem exists in all forms of human-human and machine-human interactions, because humans cannot always specify with perfect clarity the objectives they want an external party, whether human or machine, to pursue. Generally, we overcome this problem through ongoing adjustments based on trial and error, theory and learning; i.e., through deviation-reducing feedback. We also address it through institutionalization – by adopting rules and conventions designed to narrow the gap between expectations and behavior.

To move from a garden variety alignment gap to an existential threat to humanity, the doomer Computer Scientists must argue that minor deviations between the intentions of human designers and the reward and training structures of the machines will be progressively amplified until the machine(s) pursue objectives that humans have not given them and do not want. They must also assume that humans cannot intervene in this process to correct those deviations - the amplification must continue uninterrupted until it takes a dangerous turn.