Pilot projects scale up

In recent years, citizens and lawmakers have become increasingly enthusiastic about the adoption of evidence-based policies and programs. Social scientists—aided in part by an increased use of field experiments—have delivered evidence of countless interventions that positively impact peoples’ lives. And yet these programs, when expanded, have not always delivered the dramatic societal impacts promised.

Should we assume, then, that we have been too quick to embrace evidence-based policymaking?

Not at all. Instead, we should acknowledge what is missing: a proper understanding of how, why, and when promising results can be delivered at scale.

Unsurprisingly, when an intervention moves from the research setting to population-wide implementation, the magnitude of the treatment effect—or the benefit-cost estimate—might change. The change can be negative or positive and is influenced by a host of factors. In a series of new papers, my colleagues and I refer to this phenomenon as the “scale-up effect.”

To understand what causes the scale-up effect, we approach the problem through the lens of economics. By recognizing the incentives of various actors in the scientific market of knowledge creation, we identify three key ingredients to understanding the scale-up effect:

What constitutes actionable evidence? Policymakers should carefully evaluate the results of policy experiments, and should avoid scaling programs before there is sufficient evidence of efficacy. The scale-up effect becomes evident when an intervention tested in a small-scale randomized controlled trial suffers from an inference problem, such as a false positive, and then is implemented at a larger scale.

How do the properties of the population affect scaling? Threats to scalability emerge when an experiment’s participants are not representative of the policy population. A non-representative participant pool can be caused by several factors, including researcher incentives. In a competitive scientific marketplace, researchers may be more likely to choose a subject pool with a greater chance of showing a treatment effect than would a random sample or the policy population.

For example, after research found that fortified salt was successful at reducing anemia rates, a program based on that research rolled out to broader Indian populations had no effect. Why? Because it turns out the initial studies sought out adolescent women, not a representative population. Although the positive effect remained intact for adolescent women, it was absent for other groups.

How do the properties of the situation affect scaling? Threats also emerge when the experimental situation is not representative of the policy domain. Properties of the situation include specifics of the program, correct dosage, correct delivery, and implementation costs, as well as spillover effects—unanticipated consequences on participants receiving the treatment, individuals in the control group, or even people who are not participating in the experiment at all.