Even if our predictions are wrong, combining them with policy considerations is good.

At this point, I would like to go back to policy. Grant applications often ask researchers to detail broader implications, and many are very eager to emphasize that research is important if it is “policy relevant.”28 My usual response to calls to be more policy relevant is that research can only be relevant to policy objectives that are clearly specified in advance. Research may be able to tell you whether one proposed policy A is more likely to be helpful for an objective than another policy B. However, in many cases, we have multiple objectives. In some cases, policies A and B may also entail costs of a different kind than the potential benefits. In many cases, researchers are eager to talk about policy objectives, but research alone cannot tell you what your priorities ought to be or how to weigh different objectives and concerns against one another. To use a pandemic example, restrictive public health measures can reduce mortality and disease, but if they also reduce economic activity, undermine education, and increase mental health problems, what metric can we use to weigh lives saved against economic losses and loss of education? Or in Afghanistan, what should be the relative weight in US policy of minimizing active conflict, protecting women’s rights, and curtailing the cost of ongoing military operations?

In short, policy priorities must inevitably be political decisions, and researchers have no special competence here over the general public. Many people implicitly assume that everyone would be likely to accept their own views as inherently reasonable if they had the same information. There may be areas where we all would converge if we had the same information. However, knowledge problems and political priorities are distinct, and I for one have many views and preferences that I know are not widely shared. That policy must be political is as trivially true as the claim that prediction is difficult. To dismiss prediction because it is difficult is a cop-out. But so is dismissing interest in policy as something that lies outside research.

A more useful and interesting response is to think about how we can have better debates and analysis of policy. What does this mean in practice? In short, I think it is useful to separate debate about

1) policy objectives, or what we wish to achieve;

2) policy alternatives, or what one might think that we could do;

3) policy consequences, or what we think different policies may lead to; and

4) cost–benefit analysis—how do we weigh consequences of one type against each other, and different objectives against each other if there may be tension between them.

All of these points are important, but my main interest here is in (3)—namely that policy consequences are future outcomes. A claim that policy A will have consequences X is thus a prediction.

For example, in early 2022, we can find many statements on the consequences of imposing sanctions on Russia for perceived aggression against Ukraine before the invasion, or retaliation after Russia invaded, either in terms of the costs imposed or effects on Russian behavior.29 But if it is difficult to predict about the future, then it is also difficult to predict policy consequences. Still, it is common to make claims about the consequences of policies with no regard for uncertainty, which Charles Manski (2013, 2020) calls “policy prescriptions with unrealistic certitude.”

The analogy between prediction and claims about policy consequences is helpful not just in stressing uncertainty, but also in allowing us to bring to the table some of the advances in research on explicit predictions to evaluate claims about policy. If you think severe sanctions would make Russia less assertive in demands on Ukraine and the West, what are the premises and assumptions that underpin the conclusion? To evaluate predictions, they should be transparent and replicable. We need to “show our work,” in order to allow others to understand how we how we get to our conclusions.30 Only by spelling out assumptions and our degree of confidence in them can we get a clear sense of the premises for a policy as well as uncertainty and potential pitfalls.

In many cases we have competing policy advice or prescriptions—or in Manski’s terminology, “dueling certitudes.” For example, someone might assert that even severe sanctions are unlikely to be effective against Russia, if the costs are not suffered directly by leaders (e.g., Kaempfer, Lowenberg, and Mertens 2004). In such cases, looking at claims as predictions and the details in how they are produced can often help us understand the sources for the divergence. For example, do we have different conclusions because people make opposite assumptions about different things, such as the bite that sanctions would have on decisionmakers? Or do we have different objectives in the first place, such as stressing the punitive effect sanctions as an end itself versus likely effects on changing behavior? Disagreement can also arise over other features that may influence the final outcome, such as the likelihood that high energy prices and demand for natural gas will undermine willingness to engage in the most costly sanctions.31

Research and prediction alone cannot directly tell you what decisions you should make—we need to specify objectives first. However, it can help make the basis for decisions more explicit and allow for a more informed debate about policy and what people actually disagree on. People may share the same objectives and priorities, yet disagree on the consequences of proposed actions. Or people may argue for different policies and actions precisely because they disagree on the objectives themselves, or people may have similar objectives but weight relative priorities differently. If they have different beliefs about the state of the world, can we specify scenarios and assign prior likelihoods to them? What would constitute more evidence for one perspective over another as time goes by?