Free beta access. 50 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

Increasing the use of randomized pilots key to effective regulation.

Cary Coglianese 12, Professor of Law and professor of political science at the University of Pennsylvania Law School, where he is also director of the Penn Program on Regulation, “Measuring Regulatory Performance: Evaluating the Impact of Regulation and Regulatory Policy,” OECD Expert Paper No. 1, August 2012, https://www.oecd.org/gov/regulatory-policy/1_coglianese%20web.pdf

Evaluation research has a vital role to play in a regulatory environment that values learning. Now that I have presented, in Section 2 of this report, a framework for selecting indicators of regulatory performance and, in Section 3, a framework for selecting research designs to determine whether changes in those indicators can be causally attributed to regulation or regulatory policy, I turn to recommendations for bringing indicators together with research designs to provide evaluations that inform future regulatory decision making. The first recommendation discussed below is to apply in each OECD country an integrated framework that combines the analysis presented in Section s 2 and 3 of this report in order to generate substantially more decision-relevant evaluation research. The second recommendation is for each OECD country to develop institutional arrangements that will promote and support high-quality execution of research that uses the integrated framework presented in the first recommendation.

4.1 Integrated framework for evaluating regulatory performance This report’s elaboration of the essential considerations that go into selecting regulatory performance indicators and evaluation research designs has not been motivated by an academic quest for knowledge for its own sake, but rather by the desire to guide research that can inform future regulatory decision making. Evaluation research can inform decision making about a broad range of policy relevant questions, such as: Should a regulation or set of regulations (or regulatory policy) continue to remain on the books? Should a regulation or regulatory policy addressing one problem be emulated to address the same problem in another jurisdiction or similar problems in the same jurisdiction? Should effort be given to modify or seek to improve a regulation or regulatory policy? Are there particular types of regulatory instruments or tools that work better than others? If so, under what conditions?

No doubt many individuals will think they have answers to these questions. But without well-executed evaluation research behind them, such answers will have little empirical basis for winning policymakers’ confidence.

Well-executed evaluation research requires integrating two major components: indicators and research designs. Figure 7 illustrates the integration of these two components into evaluation, as well as how that evaluation generates results that, when communicated and interpreted, can support decision making. Selecting appropriate indicators and research designs, and then integrating them with each other, will often call for very case-specific judgments, depending as they do on both the purpose of the evaluation and the availability of data. Nevertheless, some generalisations may be made. The boxes for indicators and research designs in Figure 2.7 list, in priority order, what are generally the better choices for indicators and research designs.

As discussed in Section 2, indicators need to be problem-oriented, focused on the ultimate outcome of concern and on other ultimate outcomes of interest. An indicator based on net benefits will, in principle, place the impacts in terms of all of these outcomes into a common unit, allowing a full comparison both across all the outcomes as well as between different regulations or policies. For this reason, a net-benefits indicator is in principle the ideal usually to aspire toward. However, if the benefits of a rule cannot be or are not placed into monetary terms for any reason, the evaluator can rely next on cost-effectiveness. Cost-effectiveness at least takes into account both the benefits of a rule and the costs, even though it does not place them into a common unit or subtract costs from benefits. For those regulations with the same kinds of benefits, a cost-effectiveness indicator will also allow for comparisons across regulations or even jurisdictions. Should cost-effectiveness not be feasible, the evaluation can simply focus on discrete impacts, good and bad, from a regulation. These would ideally include impacts in terms of the ultimate outcome of concern, although separate indicators could be used to assess impacts in terms of other outcomes as well.

As discussed in Section 3, research designs should aspire to approximate the conditions in a laboratory experiment in order to provide confidence in causal inferences (Levitt & List, 2008). The best way to do so would be to deploy randomised experiments. Two sufficiently sized groups that are picked randomly – one that gets the treatment, the other that does not – will best ensure that observable and unobservable confounders are balanced across both groups, meaning that any statistically significant differences in the evaluation indicators between the two groups can be attributed confidently to the regulatory treatment. In the absence of a randomised experiment, the evaluation will next best be based, all things being equal, on observational studies. Of course, the problem of confounders will need to be addressed with any observational study, and depending on the situation the evaluator can choose from among several statistical methods to try to control for the effects of the confounders. However, if the large data sample needed for these statistical analyses is not available, it is possible, as a final research design, to engage in qualitative inference by designing small sample research to try to ―control‖ for confounders as much as possible, such as by conducting matched case studies.

The opportunities to increase the use of random experiments cannot be overemphasised, especially for the purposes of evaluating certain kinds of regulatory policy. If a government would like to know if a particular process for reviewing rules or engaging the public does in fact yield administrative, democratic, technocratic, or economic improvements, the best way to find out will be to randomly assign rules for application of the process. To be sure, experimentation does already take place to some degree, with regulators conducting so-called pilot projects; however, far too seldom are the selections for pilot projects conducted randomly. Pilot projects conducted on a nonrandom basis are like a nonattributional trial of a new medical device. It might appear to work well, at least against some inchoate expectations, but the true efficacy can only be determined by credible attributional research.

Sign in to recut, update, or save this card.