Combatting misinformation is key. Overwhelming scientific evidence concludes we’re psychologically hard-wired to be susceptible.
Herbert Lin 19, senior research scholar for cyber policy and security at the Center for International Security and Cooperation and Hank J. Holland Fellow in Cyber Policy and Security at the Hoover Institution, both at Stanford University, June 2019, “The existential threat from cyber-enabled information warfare,” Bulletin of the Atomic Scientists, Vol. 75, No. 4, p. 187-196, modified language indicated in brackets, ¶ indicates start of a new paragraph //chico
Research in the fields of cognitive and social psychol- ogy has formalized what Hitler knew intuitively. We now understand that human cognitive processing cap- ability is not unlimited; humans have finite cognitive resources that can be “used up” under mentally stress- ful circumstances. Findings from the same cognitive psychology that has transformed neoclassical econom- ics into behavioral economics (and resulted in three Nobel Prizes in economics) have made clear the “bounded rationality” of human thought and the simul- taneous existence in every individual of the capability to engage in two types of cognitive processing.
Specifically, heuristic dual-system cognitive theory posits that human beings have two systems for cogni- tive processing – an intuitive, reflexive, and emotionally driven mode of thought (often designated as System 1) and a slower, more deliberate, analytical mode of thought (often designated as System 2). Kahneman (2011) provides a primer on System 1 and System 2 thinking. (See Petty and Cacioppo 1986; Chaiken 1987 for other variants of dual-system cognitive theory; see Kruglanski and Thompson 1999 for a contrary view on dual-system cognitive theory.)
System 1 is designed to operate rapidly, but it can do so because it does not take account of all available information and is thus more prone to error (also called bias). System 2 operates more slowly but is more likely to take into account the available information and is less prone to error. People engaging in System 1 infor- mation processing respond more emotionally and less rationally or critically than in System 2 processing.
Most important, System 1 thinking is the default mode of thought for human beings – it uses smaller amounts of cognitive resources, relies on simple gut- based judgments, and is used more often when humans are under stress. For most situations encoun- tered in everyday life, System 1 thinking is adequate and produces mostly valid and useful outcomes, but it often fails when a situation requires complex inferences for understanding. For such situations, System 2 think- ing, which is effortful and consumptive of cognitive resources, is more often appropriate – and when indi- viduals fail to use System 2 when it is appropriate to do so, they are easily misled.
Most individuals are capable of both System 1 and System 2 thinking; thus, the important operative question is the circumstances under which they select one or the other type of thinking. Psychology has accumulated considerable evidence relevant to this question. For example, Taber and Lodge (2006) show that an individual tends to be less critical of information that is favorable to [their] position than of information that is not favorable – that is, [they are] more likely to engage in System 1 thinking for favorable information. People have a confirmation bias in their information seeking and processing behavior – they preferentially seek out information that is consistent with their beliefs and they are highly critical of (or ignore) information that contradicts their beliefs. In a meta-analysis of 91 studies, Hart et al. (2009) considered two motivations for how an individual might select information to consume – the desire to gain an accurate understanding of reality and the desire to feel validated in [their] beliefs. These two motivations conflict when an accurate understanding of reality does not validate one’s beliefs, and such a situation motivates the question of which of these motivations is more powerful. Hart et al. concluded that both motivations drive human information- seeking behavior, thus moderating each other to a certain extent, but that on balance, humans do exhibit a tendency towards the validation of their beliefs. People are also subject to belief perseverance (a.k.a. a continuing influence effect) – a cognitive bias through which individuals do not revise beliefs based on erro- neous information even when they know for sure that such information is erroneous (Lewandowsky et al. 2012).
Maintenance of an individual’s social identity is an important influence on his or her invocation of System 1 or System 2 thinking. Evidence suggests that indivi- duals tend to adopt the views of the peer groups that are most salient to them, even if the “objective” or “factual” information available to them contradicts those views. (Asch 1951 performed the classic “confor- mity experiments” that demonstrated this phenomenon in the early 1950s.) Uncritical System 1 thinking is active in processing information that is consonant with the beliefs and attitudes of those peer groups. Critical and skeptical System 2 thinking is active in processing infor- mation that is dissonant to those groups’ beliefs. These effects (that individuals tend to accept salient group norms) are even more pronounced in an anonymous environment, such as that which characterizes much online interaction (Postmes et al. 2001).
Lastly, there is evidence that emotion and motivation affect cognition. For example, people who are angry tend to rely more heavily on simple heuristic cues (sug- gestive of System 1 thinking) than those who are not angry (Bodenhausen, Sheppard, and Kramer 1994). Individuals are more likely to stereotype people (a form of System 1 thinking) when that stereotype is consistent with their desired impression of those people; conver- sely, when the stereotype is inconsistent with their desired impression, individuals tend to inhibit the use of this stereotype (Kunda and Sinclair 1999). Negative emotions (such as those induced by the receipt of infor- mation incongruent with a person’s prior beliefs) can improve the ability of a person to reason logically, thus enabling [them] to negate or discount that informa- tion (Goel and Vartanian 2011).
In the new information environment, exploitation of human cognitive architecture and capabilities – which are largely unchanged from what existed millennia ago – provides the 21st century information warrior with cyber- enabled capabilities that Hitler, Stalin, Goebbels, and McCarthy could have only imagined. By exploiting cogni- tive limitations, the perpetrators of cyber-enabled infor- mation warfare have learned to exacerbate prejudices, biases, and ideological differences; to add heat but no light to political discourse; and to spread widely believed “alternative facts” in advancing their political positions.
Russian interference in the 2016 US presidential elec- tion has dominated news headlines ever since. But interference by authoritarian countries in the elections of democratic states – as undesirable and threatening as it may be – is hardly the only negative consequence of cyber-enabled information warfare. The problems of nuclear war and climate change are hard enough to solve even when well-intentioned, well-informed par- ties on all sides share a basic understanding and knowl- edge of the relevant facts. Yes, they may have different values and different priorities, may act under different constraints, and be able to bring to bear different levels of resources to these problems.
But without shared, fact-based understandings of the blast, thermal, and radiation effects of nuclear explosions, what hope is there for national leaders to reach agreements to reduce the threat of nuclear holo- caust or to make good decisions about nuclear weap- ons use in times of crisis? Without shared, fact-based understandings that rising atmospheric carbon dioxide concentrations caused by human beings result in cor- responding increases in global temperature and cli- matic disruption, what hope is there for national leaders to reach agreements to begin serious efforts at decarbonizing their economies?