Debating policy forecasts in a competitive tournament format anchored by fair adjudication encourages epistemic humility and cognitive flexibility that spills outside debate—it’s unique because broad under-forecasting is breeding societal dogmatism now

We assume that the future is not perfectly knowable by any political faction—and that political beliefs are better framed as continuous probabilities than as off-on, dogmatic certainties. Although this assumption sounds uncontroversial, it is widely ignored in the heat of political controversies. Content-analytic studies of election campaigns and of debates in elite legislative bodies have found a widespread tendency to express political opinions in sharply dichotomous fashions (Hopkins and King, 2010, Tetlock, 1981, Tetlock et al., 1984). Good-bad, categorical rhetoric is common, and nuanced probability judgments are rare. Americans are more polarized now than at any time in decades (Pew Research Center, 2014, Pew Research Center, 2017). Many of us appear to be Manichaean thinkers to whom probabilistic thinking does not come naturally (Kahneman, 2011). All of which raises a question that is part descriptive and part normative: How feasible is it to induce people to treat their political beliefs as testable probabilistic propositions open to revision in response to dissonant as well as consonant evidence and arguments?

A survey of the psychological literature suggests the grounds for pessimism out-number those for optimism. People are often too quick to jump to conclusions from fragmentary evidence (Fiske and Taylor, 1991, Ross, 1977), too slow to update their views in the wake of disconfirming evidence (Lord, Ross, & Lepper, 1979), too tempted to apply weak “can-I-believe-this?” tests to agreeable claims and tough “must-I-believe-this?” tests to dissonant claims (Belsky & Gilovich, 2010), too prone to over-confidence (Moore & Healy, 2008; Lichtenstein, Fischhoff, & Phillips, 1982) and over-estimation of how much others share their views (Mullen et al., 1985, Ross et al., 1976), and too fond of seeking out the company of like-minded others (Sunstein, 2004) and, once in their company, of embracing increasingly extreme views (Myers and Lamm, 1976, Sunstein, 2007). Whether working individually or in groups, people are often susceptible to self-justifying cycles of reasoning that make them progressively more self-righteous and contemptuous of the competence and morality of the other side (Haidt, 2012). In this gloomy view of citizen reasoning, we are “naïve realists” (Ross, in press) who are fated to be “prisoners of our preconceptions” (Tetlock, 2005).

There are however pockets of experimental evidence that give us guarded grounds for optimism that, under well-defined conditions, people can shift into more flexible modes of thinking and treat their political attitudes less as dogmas and more as testable propositions. Debiasing studies have shown that careful engineering of the questioning process and social context can induce people to become more aware of how little they know, more attuned to alternative ways of looking at the same reality, and more skilled at constructively criticizing their own views as well as those of others—all of which are widely seen as essential cognitive virtues of democratic citizenship (Gutmann & Thompson, 2009). We divide the existing work on debiasing citizen reasoning into three categories.

First, building on Rozenblit and Keil’s (2002) work on the illusion of explanatory depth, Fernbach et al., 2013, Sloman and Fernbach, 2018 have shown that asking people to explain complex policies, such as transitioning to a single-payer health care system, makes people appreciate the depth of their ignorance which in turn causes them to moderate their policy preferences. It is essential though that the requests be explanations for “how” policies work, not “why” one supports or opposes them—and that the requests fall in utilitarian rather than deontic domains, topics on which people feel that merely invoking feelings or hunches would not count as a sufficient rationale.

Second, building on the work of Ross (Lord et al, 1979) and Koriat et al., 1980, Hirt and Markman, 1995 showed that directing people to “consider the opposite” and “consider an alternative strategy” reduces their confidence in the focal hypothesis and allows them to entertain divergent scenarios. Arkes, Faust, Guilmette and Hart (1988) found that this strategy helped neuropsychologists asked to read a case study and assign confidence judgments to three diagnoses. Those who listed at least one piece of evidence consistent with each diagnosis before assigning confidence judgments were less prone to the hindsight bias than those who were told the cause and then asked to imagine the confidence ratings they would have assigned to each causal candidate.

Third, building on classic symbolic-interactionist ideas about perspective taking in social interaction (James, 1890, Mead, 1934), many studies have shown that carefully choreographed forms of accountability can induce people to become more “integratively complex” thinkers, that is, people who respect and balance opposing views and resist biases such as belief perseverance and over-confidence (Lerner and Tetlock, 1999, Tetlock and Kim, 1987). But researchers need to ensure that participants know they are accountable before voicing their opinions, and researchers also need to ensure that participants cannot easily guess the opinions of those to whom they must answer.

In brief, researchers need to create pre-decisional accountability to an unknown audience—a type of accountability that we do not often encounter in the world but that blocks off cognitively simpler coping strategies, such as attitude shifting toward the evaluative audience (Cialdini, Levy, & Petty, 1976) and defensive bolstering of past attitudinal commitments (Festinger, 1957). This sweet-spot form of accountability boosts the integrative complexity of reasoning in numerous studies (Lerner & Tetlock, 1999). People seem to engage in a coping response known as pre-emptive self-criticism, in which they concede legitimacy to a wider range of views (evaluative differentiation) and try to understand how reasonable people can see the same events differently (Schroder et al., 1967, Tetlock, 1983). Having done so, people are also likelier to moderate their beliefs.

Across all three lines of investigation, it is striking how careful “debiasing” investigators need to be in designing their interventions to pry open otherwise closed minds. Much hinges on who is requesting explanations for which opinions, how they word their requests and why people think they are being asked.

We explore a complementary approach to debiasing political judgment: forecasting tournaments that challenge people to assign accurate probability estimates to a wide range of events. Tournaments are inherently multifaceted manipulations that have arisen in response to the practical demands of real-world organizations to provide policy-makers with timely probability estimates of the consequences of options (Tetlock and Gardner, 2015, Wolfers and Zitzewitz, 2004). When we inspect each facet however, we find close parallels between their intended functions in tournaments and the goals behind debiasing manipulations developed by experimental psychologists.

Tournaments put participants in an unusual social world in which all three debiasing factors mentioned above are at work. First, players are strictly accountable for the accuracy of their views, not for offering views that play to the prejudices of the like-minded as is common in the political world. Wishful ideological thinking will quite directly translate into worse performance. Second, tournaments pressure people to acknowledge opposing arguments in the news, as well as gaps in their knowledge, both of which should have effects that parallel those observed in the illusion-of-explanatory-depth research: reduce confidence to more realistic levels and moderate affective preferences. Third, if one’s goal is to do well in a political forecasting tournament, one is well advised to engage in perspective taking, a defining property of actively open-minded thinking (Baron, 2008) and an empirical correlate of top performance in tournaments (“superforecasters”—Tetlock & Gardner, 2015).

These parallel lines of work—laboratory debiasing and forecasting tournaments—raise two questions for democratic theory: (a) if political opponents were to participate in forecasting tournaments focused on issues of the day, would they learn to be more circumspect in their beliefs?: (b) if so, would that process also reduce the extremity of their political attitudes, even political preferences with no logical connection to the probabilistic forecasts elicited in the tournament?

We present evidence for conditional optimism. Forecasting tournament, participants must learn to cope with the “culture shock” of moving from the hurly-burly of the political world, in which polemics dominate, to the pure-epistemic-accountability world of tournaments. Tournament organizers can help with this adjustment by giving participants guidance on the larger purpose of the exercise, on how to translate inchoate hunches into precise probability metrics and on how the scoring rules convert probability judgments into accuracy scores. In sharp contrast to the vague-verbiage predictions so common in high-visibility, social-media debates in op-eds and blogs, forecasting tournaments do not give bonus points for witty put-downs. They incentivize forecasters to do one thing: getting the best “Brier scores” (Brier, 1950) by being faster than others at putting higher probabilities on things that happen and lower probabilities on things that do not while avoiding the steep scoring penalties that accrue to too hastily declaring outcomes “certain” or “impossible” (Mellers, Stone, Atanasov, et al., 2015). This trade-off is tricky. It requires balancing caution and courage (Tetlock, 2005), where caution refers to skill at making well-calibrated probability judgments that match relevant base rates (e.g., events assigned 70% probabilities happen about 70% of the time) and courage refers to skill at decisively discriminating between signals and noise, events and non-events, as the evidence warrants. Forecasting tournaments are cognitively demanding.

1.1. The Good Judgment Project The largest scale, political-forecasting tournaments to date have been sponsored by IARPA or the Intelligence Advanced Research Projects Activity. From 2011 to 2015, IARPA invested tens of millions of dollars in five university teams, each of which competed to invent the best methods of eliciting and aggregating large numbers of human forecasts in tournaments focused on geopolitical events. Our team, the Good Judgment Project, won these contests. Psychologists and statisticians working together constructed aggregate forecasts on the “right side of maybe” on more than 85% of all daily forecasts on over 500 questions, outperforming even professional intelligence analysts who made predictions on the same questions and had access to classified information (Goldstein, Hartman, Comstock, & Baumgarten, 2018). The Good Judgment Project conducted experiments by randomly assigning forecasters to conditions that tested hypotheses about the drivers of accuracy. We discovered five such drivers: (a) recruitment and retention of better forecasters; (b) cognitive-debiasing training; (c) more engaging work environments, in the form of collaborative teamwork and prediction markets; and (d) better aggregation rules for distilling the wisdom of the crowd (the log-odds-extremizing algorithm of Baron et al., 2014, Satopaa et al., 2014, Satopaa et al., 2014). But the most potent driver was to skim off the top 2% of forecasters at the end of each tournament year and place them in elite, superforecaster teams. Despite skeptics’ predictions to the contrary, superforecasters largely resisted regression to the mean in follow-up years (Mellers, Stone, Murray, et al., 2015). Superforecasters learned to make well-calibrated, high-resolution probability judgments with quite impressive consistency (Moore et al., 2016, Tetlock and Gardner, 2015). They figured out how to treat uncertainty in a nuanced fashion and use more distinctions on the probability scale to express their beliefs. These distinctions turned out to be meaningful. When we examined the loss of accuracy with counterfactual analyses in which forecasters were assumed to use fewer probability distinctions by rounding their predictions to the nearest 10% or 20%, accuracy scores based on rounded forecasts were worse than accuracy scores based on the original forecasts. Rounding forecasts produced a loss of information that took the largest toll on the most accurate participants– superforecasters (Friedman, Baker, Mellers, & Zeckhauser, 2017). 1.2. The spillover hypothesis: beliefs and attitudes Our main goal is testing the hypothesis that the process of making subjective-probability forecasts in tournaments has spillover effects on attitudes toward controversial political issues. From a formal philosophical perspective, these two classes of variables are clearly logically distinct. Forecasts are beliefs about matters of (future) fact, whereas policy attitudes are ultimately value judgments about what society ought to do. But of course that does not imply that fact-grounded forecasts and value-grounded attitudes must also be psychologically distinct. Much hinges on one’s preferred cognitive-affective framework for representing the connections between beliefs (often operationalized as probability judgments in the attitude change literature) and attitudes (operationalized as evaluative or affective judgments). For instance, viewed from the standpoint of attitude theories that stress the primacy of affect and treat cognition (i.e., beliefs) as post hoc justifications for feelings (Haidt, 2001, Zajonc, 1980), our spillover hypothesis is unrealistically optimistic. Tails don’t wag dogs. Changing policy affect is far likelier to change policy beliefs than is changing the beliefs to change attitudes. But viewed from the standpoint of cognition-driven theories that depict attitudes as derivative products of expectancy-value calculations, as in Fishbein and Ajzen’s (1975) theory of reasoned action and in classical decision theory, the spillover hypothesis is reasonable as long as the attitudes are woven into networks of relevant beliefs. If so, when the beliefs moderate, so too should the relevant attitudes.

Neither of these theoretical camps would however predict the strong version of the spillover effect that we are proposing: namely, that tournaments can moderate attitudes by changing how people think, a cognitive-stylistic shift in which the more flexible and analytic style of thinking about beliefs encouraged by tournaments spills over into how people make “should” or attitudinal judgments on topics with close-to-zero expectancy-value linkage to the probabilistic beliefs they endorsed in tournaments.

To operationalize this idea, imagine we ran a study in which the forecasting questions focused on exactly the same topics as the attitude questions—and we discovered, say, that the (humbling) process of trying to forecast in which cities charter schools boosted test scores caused people to moderate their policy attitudes about charter schools. Few observers would be surprised. But the current design provides a tougher test. There are no strong connections between the forecasting questions and the policy-attitude questions asked at the end of the forecasting year. We test the hypothesis that the cognitively demanding process of participating in a tournament—of having to balance evidence to put well-calibrated probability estimates on, say, third quarter GDP growth or presidential popularity—has a spillover effect when people take normative/”should” attitude positions on unrelated issues such as the minimum wage or voter-identification laws or U.S. policy toward Iran.

Tournaments require forecasters to transform their hunches into numbers that roughly conform to the axioms of probability, a nontrivial task. When they do internet searches to discover what high-profile pundits say, they get little help because the “predictions” advanced in such debates are rarely sufficiently explicit to be testable. After all, elites need to preserve their long-term credibility so they tend to avoid numerical probability judgments that could later be mocked as embarrassingly far off (Tetlock & Gardner, 2015). It is safer to make vague-verbiage forecasts, such as “x might well happen” or “there is a distinct possibility of y” that psychological research has shown comfortably straddle both sides of 50/50 or “maybe” (Wallsten, Budescu, & Zwick, 1993). If an event happens, one can say “I warned you it was a distinct possibility” and if the event never happens, one can shrug it off by saying “I merely said it was possible.”

To avoid this problem, forecasting tournaments press people to make their beliefs explicit and do what Nate Silver, the founder of the 538 website, routinely does: attach specific numerical probabilities to possible futures. After the 2016 US Presidential election, Silver was chastised for falling on what appeared to be the “wrong side of maybe.” He had placed a 70% probability on a Clinton presidency. Was he wrong? Drawing the correct conclusion requires an understanding that: (a) 70% might have been the best forecast possible at the time; (b) the poll aggregation techniques on which Silver relies were well calibrated, which means that when Silver makes probabilistic predictions of events, those events tend to occur roughly 70% of the time. They will “look wrong” about 30% of the time (Lichtenstein et al., 1982).

Unfortunately, the vague-verbiage forecasting habits that confer political safety on pundits also make it extremely difficult, arguably impossible, to become a more realistic appraiser of the probabilities of real-world events. Without measurement of forecasting accuracy, learning can’t take place. To invoke an old Wittgenstein aphorism, the limits of our language are the limits of our world. It is hard to figure out whether probability judgments of 70% are well-calibrated if one is unsure when one made 70% forecasts and unsure of what happened when one did. The potential mind-opening power of forecasting tournaments comes from the fact that they pressure us to recognize the limits of our knowledge and focus on the world as it is, not as we wish it to be. They expose us to the same criticisms that Nate Silver experienced -- being on the “wrong side of maybe.”

Forecasting tournaments push people out of their vague-verbiage comfort zone and ask them to quantify uncertainty in as granular a fashion as the problem permits. It is this more tentative and granular mode of thinking inculcated in forecasting tournaments that we hypothesize can spill over into unrelated attitude judgments outside the tournament.