Free beta access. 50 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

SHAM PILOTS---the perm sets a precedent for ‘sham experiments’ that aren’t allowed to shape program implementation.

Kristen Underhill 19, Associate Professor of Law, Columbia Law School, “Broken Experimentation, Sham Evidence-Based Policy,” 38 Yale L. & Pol'y Rev. 150, Fall 2019, Lexis

My goals in this Article begin with an effort to place current interest in EBPM in context, examining the theoretical appeal and advantages of EBPM, the limitations of evidence and its uptake in government decision-making, and a brief history of the rise and uncertain trajectory of EBPM at the federal level. The second aim of this work is to identify ways in which EBPM can go wrong. Building on recent examples in “evidence-based” efforts, I develop an original taxonomy of specific pathologies, ranging from the neglect and deliberate undermining of research (broken experimentation), to the distortion of evidence (a form of sham EBPM), to the disingenuous use of selective evidence to de-fund entire program categories (ratcheting). A third goal of this Article is to illuminate the stakes, potential costs, and possible benefits of sham EBPM, including implications for welfare, efficiency, scientific legitimacy, the production of knowledge, human subjects enrolled in governmental studies, and potential positive externalities.

Building on these arguments, my final aim herein is to propose remedies for the current pathologies in the production and uses of evidence. I begin, however, with a call to set a more realistic scope for the policy areas where we might hope to introduce research evidence productively. Some areas are more prone to EBPM pathologies than others, particularly problems of evidence distortion, weakening scientific legitimacy, and burdens on human subjects who enroll in research with no chance of policy payoff. But where EBPM is possible—where regulators or legislators are supportive of efforts to use empirical evidence productively—I propose that innovations such as ex ante policy commitments (e.g., pre-commitments contingent on research findings), evaluation mandates, and rules for evaluation transparency may avoid some of the pitfalls that currently stymie EBPM.

This Article proceeds in the following Parts. Part I begins with the merits and limits of evidence-based policy-making, a process that includes both governmental decision-making and subsequent evaluation of those choices. In its ideal form, EBPM improves the likelihood that governmental actions will achieve their own ends, making it agnostic to policy preferences. EBPM asks regulators and legislators to identify, synthesize, and interpret the empirical evidence informing each decision; to make decisions in the direction favored by the evidence; to evaluate the outcomes of decisions; and to update decisions as informed by new evidence. This Part will provide a brief overview of EBPM’s inroads into regulation and legislation.

Part II turns to ways in which EBPM can fall so far short of the ideal as to be “sham,” with costs for both the legitimacy and the practical consequences of government choices. I focus first on common pathologies in governmental efforts to generate evidence (broken experimentation), and then turn to visible pathologies in the application of evidence (sham EBPM). Here I consider four central pathologies of sham EBPM: technical breakdowns in the uptake of evidence, distortion of evidence, terminal or symbolic experimentation without broader adoption of results, and ratcheting cuts to program categories. The central concerns of this Part are deep-seated methodological flaws in evidence generation, as well as irreparable gaps in the feedback loops between evaluation and policy decisions.

Part III will consider the ethics and costs of broken experimentation and sham EBPM, including consequences for allocation of resources, the legitimacy of science, the interests of individual human subjects, and the communities those subjects represent. I will argue here that, despite some positive externalities, sham EBPM can cause harm, including wasting public resources, misleading the public, eroding scientific legitimacy, and threatening the dignity of human subjects.

In Part IV, I seek to reclaim EBPM and governmental experimentation by identifying conditions where program and policy evaluations are likely to be both rigorous and effective in shaping policy. By extension, I will also identify conditions where evidence is likely to be distorted, misleading, or irrelevant. Evaluation should be both more and less used than it currently is—more so in contexts where it would be effective, but perhaps less so when there is reason to believe it will skew or contribute little to decisions. Where regulators and legislators do build evaluation into their policy choices, they have a range of legal options to avoid sham evaluations—both by setting explicit terms for study design, but also by taking affirmative, ex ante, and perhaps binding steps to define how results will affect subsequent policy choices.

Sign in to recut, update, or save this card.