Mathematical models disprove threat inflation.

Our findings warrant tempered optimism about the quality of strategic intelligence forecasts. The forecasts fared exceptionally well in terms of discrimination and calibration, and these results were not particularly sensitive to case exclusions. The results provide a stark comparison with Tetlock’s (17) findings. Whereas the best political forecasters in his sample explained about 20% of the outcome variance, forecasts in this study explained 76% of the variance in geopolitical outcomes. Likewise, the mean absolute deviation from perfect calibration (the square root of the calibration index) was 25% smaller in our study than in Tetlock’s, reflecting better calibration. Experience had no effect in his study, but analytic experience led to a practically beneficial improvement in discrimination in this study. Not only did senior analysts discriminate better than junior analysts, they produced 68% of the forecasts despite constituting less than half of the analyst sample (Materials and Method). Finally, whereas experts in Tetlock’s study showed overconfidence, forecasts in this study were underconfident. Although unbiased confidence is optimal, the cost of overconfidence in intelligence will usually outweigh that of underconfidence. Underconfident forecasting occurs when analysts discriminate better than they think they can. Although underconfidence reduces informational value for decision makers by expressing forecasts too timidly, it signals good judgment. Overconfidence, in contrast, indicates that forecasts are communicated with unwarranted certainty, implying that they are more error-prone than experts predict.

Fig. 4. Calibration curves before and after recalibration to t*.

Several of the findings suggest that intelligence producers adhere to a professional norm of promoting informativeness while avoiding overstatement. First, there were few uninformative forecasts near maximum uncertainty (probabilities of 0.4–0.6). If analysts were only concerned with playing it safe, we would likely have seen a bulge rather than a trough in that region. Second, underconfidence was more pronounced when the issues were relatively complex and more important for policy makers. These variations are consistent with a norm of caution, which both difficulty and consequence of judgment should heighten. Indeed, the effect of difficulty may otherwise appear surprising given that harder problems usually produce greater overconfidence (30). As well, this pattern (greater underconfidence for harder forecasts) does not support the explanation that underconfidence in this study was mainly due to easy forecasting problems. A norm of caution may serve the intelligence community well in the absence of systematic feedback on the forecasting quality. With such feedback, adjustments to cautionary normative pressure could lead to crisper indications as analysts and their managers become aware of characteristics such as good discrimination (5). That would, in turn, better serve the aim of reducing uncertainty in the mind of the decision maker (2).

A deeper account of the present findings might trace normative pressures for cautious, yet informative assessments to analysts’ accountability to multiple, skeptical audiences. Accountability pressure has been shown to reduce overconfidence in predicting personality attributes (42), reduced overattribution bias (43), deeper information processing (44), and better awareness of the informational determinants of one’s choices (45). As Arkes and Kajdasz (46) have already noted about a preliminary summary of our findings presented to the National Research Council Committee on Behavioral and Social Science Research to Improve Intelligence Analysis for National Security, the strategic analysts who made the forecasts studied here had much more severe accountability pressures than experts in Tetlock’s (17) study and in other academic research. Experts in such studies know that their forecasts will be anonymously analyzed in the aggregate. In contrast, analysts are personally accountable to multiple, skeptical audiences. They have to be able to defend their assessments to their managers. Together with those managers, they need to be able to speak truth to power. Analysts are acutely aware of the multiple costs of getting it wrong, especially on the most important issues. The flipside of accountability pressure is feedback, which may also benefit accuracy. Given that analysts assessments must be vetted by managers and may receive input from peers, they may also benefit from the underlying team process, as has recently been verified in a controlled geopolitical forecasting tournament (47).