The CP is an opportunity cost!
Colleen V. Chien 19, Justin D'Atri Visiting Professor of Business Law, Columbia Law School; Professor, Santa Clara University School of Law; 2013-2015 White House Senior Advisor, Innovation and Intellectual Property, “Rigorous Policy Pilots: Experimentation in the Administration of the Law,” 104 Iowa L. Rev. 2313, July 2019, Lexis
Rigorous piloting involves selecting a test strategy that is appropriate to the question being asked. A “mixed methods” strategy of getting feedback, e.g., through surveys, interviews, case studies, and statistical (causal and non-causal) approaches is often best. Regardless of the specific method of evaluation, though, each of the other parts of specifying a pilot: articulating the question to be addressed, a theory of change, the evidence, how to measure it, and matching resources to the pilot, will require attention and in some cases, consultation with stakeholders, cost and convenience trade-offs, and implementation risk. In some cases, developing the right metric will be transformative, in others, framing the question correctly can unlock greater learning.
But when the question is, “did policy X have Y effect,” experimental or quasi-experimental approaches are understood to yield the best information. Volumes have been written about appropriate methods of evaluations. But also worth underscoring is what government pilots have revealed about the value of rigor, and the wasted opportunity associated with the lack thereof.
Section 1115 of the Social Security Act allows states that receive approvals from HHS to test and evaluate new approaches for delivering Medicaid services using funds that would not otherwise be eligible. Taking advantage of these waivers, by 2016, nearly three-quarters of states implemented at least part of their programs under demonstrations. To evaluate what had been learned, the Government Accountability Office (“GAO”) reviewed demonstration projects in eight out of 15 states. The results were not good. The GAO concluded that a lack of care in how the evaluations were carried out, reported, and shared compromised their usefulness. The lack of control groups in four out of the eight states, insufficient sample sizes and response rates for surveys, and the inability to isolate the impact of the Medicaid interventions from other changes limited what could be concluded about the effectiveness of demonstrations. Rather than resource deficits being the problem—in 2015 alone, states spent over $100B, or one-third of Medicaid program expenditures on “demonstrations”—the problem seems to have been systemic attention deficits, to specifying upfront how evaluations could meaningfully be done, for example by implementing them in a way that would support experimental or quasi-experimental evaluation, as well as, perhaps a lack of a bona fide intent to test. These “significant limitations . . . affected their usefulness in informing policy decisions.”
This description suggests that the lack of close alignment between those requesting and those implementing pilots can drastically reduce their value as a learning exercise. One way to avoid this result is by ensuring that interventions are grounded, or at least framed, in terms of the lived experiences of those required to implement them. Presenting “rigorous pilots” as tools for advancing and testing assumptions and theories developed from the “bottom up” can distinguish them from “top-down” evaluation mandates that are not well integrated into agency decision-making.