Free beta access. 50 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

CONTROL GROUP---premature uniform application means there’s no baseline for comparison---this makes programs ineffective.

Colleen V. Chien 19, Justin D'Atri Visiting Professor of Business Law, Columbia Law School; Professor, Santa Clara University School of Law; 2013-2015 White House Senior Advisor, Innovation and Intellectual Property, “Rigorous Policy Pilots: Experimentation in the Administration of the Law,” 104 Iowa L. Rev. 2313, July 2019, Lexis

The range of open questions in law and policy that can be addressed by pilots are wide-ranging. They include, for example, the question of how giving a universal basic income ("UBI") to individuals impacts their well-being and employment. In 2017, the Finnish government began tracking the outcomes of 7,000 unemployed citizens, about a third to which it gave $ 600 a month, [*2317] to address this question; 14several U.S. jurisdictions are planning their own tests. 15Also unknown is how the administration of patent law and issuance of quality patents best advance the Constitutional goal of "promoting the progress of science and useful arts." 16Over the last decade, the United States Patent and Trademark Office ("USPTO") has run numerous pilots to address this question. 17As the D.C. Circuit has said, "there are some situations in which, "a month of experience will be worth a year of hearings.'" 18While experiments or pilots can address a variety of questions pertaining, e.g., to a policy's feasibility or stakeholder reactions, when the question concerns whether the policy has caused an observed outcome, a controlled trial that provides the treatment to one part of population, withholds it from another, and then compares the difference in outcomes provides the best insight. Such rigorous experiments can be designed, in turn, with or without randomization, and to vary the rules or laws that apply or to approximate such changes. A pilot that limits to whom a rule or policy applies constrains the risk but not the learning. As Justice Brandeis famously dissented, "a single courageous state may ... serve as a laboratory; and try novel social and economic experiments without risk to the rest of the country." 19

Within today's highly partisan political landscape, the need for quality evidence about the performance of law and policy presents a rare opportunity for agreement, on the methods if not the values or agendas advanced by evidence. Calls for "smarter use of data and evidence ... to orient decisions and accountability around service and results" 20and "an aggressive management agenda ... that delivers smarter, more innovative, and more accountable government for citizens ... . by applying ... evidence about what [*2318] works" 21have appeared in President Trump's and President Obama's management agendas, respectively. The passage of the Foundations for Evidence-Based Policy Making Act of 2018 ("2018 Evidence Act"), besides showing that Democrats and Republicans actually can agree on something, will consolidate and coordinate statistical expertise for assessment among rank and file bureaucrats across the federal government. 22

But while "adequate and well controlled studies" have long been required in certain contexts, 23experiments with laws and rules have a history of being constitutionally and legally suspect. As the Supreme Court stated in Truax v. Corrigan, "the Constitution was intended - its very purpose was - to prevent experimentation with the fundamental rights of the individual." 24

Rigorous pilots that provide a benefit to some, but not other, members of a population would appear to offend basic notions of equal protection. 25Imposing a regulatory burden on a subset of regulated entities likewise sounds in unfairness and risks running afoul of the Constitution's due process guarantees. Applying a rule randomly to one set of entities but not another would appear to some to present just the kind of "arbitrary, capricious" agency action judicial review was designed to set aside. 26

How rigorous testing can support evolution of the law is also not widely understood. By operating within existing legal frameworks, experiments can obscure more salient questions, including whether or not the framework is the right one in the first place. Rigorous experimentation is less flexible and adaptable than "democratic experimentalism" with which it shares some features. 27While Congress may feel empowered to take bolder steps when laws are framed as reversible and temporary, requirements of rigor can constrain agencies, feeding anti-regulatory impulses. Experimental evidence can be weaponized and politicized in the same way non-experimental evidence can. 28

[*2319] And yet, when compared to doing nothing or engaging in unreflective action, the benefits of rigorous experimentation become clearer: Rigorous pilots can prime legislative and policy-making efforts by underscoring the importance of evidence and prompting a discussion about the metrics that matter. Rigorous pilots can be used to test legal theories, and point out legal infirmities, provided that is the motivation. Further, among information sources, rigorous pilots are distinct in their ability to generate experiential feedback on the operation of law and policy.

But while rigorous policy pilots can thus be powerful tools, so far, they have been underused tools for informing the development of law and policy. In part this is due to a perception that certain types of rigorous pilots are possibly illegal. Citing a litany of difficulties, a recent recommendation of the Administrative Conference of the United States 29concluded that, "legal, policy, and ethical challenges ... may mean that regulatory agencies should use randomized study methods only under limited circumstances." 30But rigorous pilots have also been underused because of practical obstacles to designing and carrying them out, and a lack of awareness of the value they can add.

This Essay attempts to chip away at perceived legal, institutional, and informational barriers to the use of rigorous policy pilots. While informed by the growing academic interest in experiments, 31it primarily draws upon perhaps the single most credible source for addressing the legality, importance, and mechanics of structured pilots by the federal government: the federal government itself. Based upon caselaw, legislative, and agency accounts of government experimentation, it finds that from being illegal, hard, and rare, rigorous government pilots are presumptively legal, feasible, increasingly common, and in many cases, worthwhile, and proposes a framework for proposing rigorous policy pilots based on these experiences. A companion online appendix, using the patent system as a case study, applies [*2320] this framework to suggest rigorous policy pilots that the USPTO could implement to support patent law-and policy-making.

Part II reviews relevant case law and finds that rigorous policy experiments raise no particular legal barriers to agency action. Many pilots will be uncontroversial because they do not require the mandatory, selective application of law or policy to the public. But pilots that do, including through randomization, have by and large been found by courts to pass constitutional muster on the basis that they further learning and other legitimate government objectives. The requirements that the Administrative Procedure Act ("APA") imposes will also vary depending on the design and features of a pilot but should not present special procedural hurdles to rigorous pilots nor elevated substantive hurdles to the use of the evidence they generate.

Having established in Part II that rigorous policy pilots are presumptively legal, Part III draws upon decades of rigorous agency experiments to show that, while not necessarily easy, rigorous policy pilots are both feasible and worthwhile in a variety of contexts. It nests lessons that can be learned from these experiences into a basic framework for proposing a rigorous policy pilot, "MATTER," that comprises the following steps: attending to questions that matter, relevant authority, the underlying theory of change, testing protocol, evidence and resources. 32It aspires to audiences of not only policymakers but also those engaged in forming policy recommendations, on the theory that framing new policy ideas as pilots to learn from, not just new policies to adopt, can increase their uptake. Part IV concludes, and an online companion appendix applies this framework to the patent system to suggest several pilots that the Patent Office could try.

II. Rigorous Policy Pilots Are Presumptively Legal

Every day, companies are developing prototypes and carrying out structured tests to find effective treatments 33and increase consumer engagement. 34Why, to date, hasn't rigorous piloting been widely used to develop and guide the evolution of federal law and policy, improving their effectiveness for the American people? Though there are many reasons, the [*2321] robustness of available evidence concerning the effectiveness of law and policy is not one of them. The fragility of government feedback loops is well-documented and structural, stemming from, inter alia, the absence of competition, 35the anecdotal and fragmentary nature of litigation, short democratic terms and broad Congressional mandates, and the prioritization, in public policy, of the urgent over the important. 36The dynamics of capture and the non-participation of impacted entities in processes like notice-and-comment and town halls, or what could be called the "turn-out problem" in lawmaking, limit the quality of feedback. 37

Against this backdrop, the promise of objective, real-world data on the performance of law or policy from a pilot might seem irresistible. However, for the average federal agency contemplating an experiment that will selectively burden or benefit members of the public, Constitutional, APA, and basic fairness considerations loom large. 38The act of designating treatment and control groups in effect "allows ... officials to pick and choose only a few to whom they will apply legislation," a practice condemned by Justice Jackson in Railway Express Agency, Inc. v. New Yorkas contrary to the Constitution. 39Random selection can feel "arbitrary-and-capricious" and potentially unlawful. Pilots that induce capital-intensive investments or that specify or result in differential levels of health or safety protection may, worryingly, place one set of regulated entities or members of the public at a disadvantage relative to another. 40

The case for caution in government experimentation is historically grounded. In the 1970's, revelations about the withholding of effective treatments from African-American men with syphilis in Tuskegee, Alabama, by the U.S. Public Health Service to complete a government experiment set [*2322] off a chain of events that led eventually to the Common Rule, 41a set of ethical and legal commitments that govern federally-funded research on human subjects. 42In their concurrence in United States v. Stanley, a case involving the Army's unconsented experimentation to test the impacts of lysergic acid diethylamide ("LSD") on servicemen, Justices Brennan, Marshall, and Stevens rebuked "the Government of the United States [for] treatingthousands of its citizens as though they were laboratory animals." 43But even less dramatic examples can give pause. Would you want to be the unemployed person that does not receive a universal basic income? 44As a small business owner, how would you feel about being included in a pilot that subjects your patent application, but not that of others, to intensified scrutiny? 45

How can desires for fairness and quality information be squared? This Part considers the Constitutional and APA requirements that apply to agency policy pilots. Before doing so, however, it is important to keep in mind that the range of possible pilots an agency can run is as vast and varied as the scope of agency action. What makes a pilot rigorous is that it uses "well-designed and well-implemented methods tailored to the questions being asked." 46As such, the design of each pilot will depend highly on context and motivation, with implications for its legality. When the relevant question concerns whether a policy change is effective (rather than just feasible or popular, based on its uptake), a rigorous approach will require the identification of a control and treatment group, which can - but does not necessarily - raise equal protection and fairness concerns. Below I describe in simplified terms, several major classes of controlled experiments before discussing their likely treatment under the law (from least to most legally challenged), providing examples from the universal basic income ("UBI") and patent pilots referred to in the introduction to illustrate each concept.

Natural Experiments: In some cases, "control" and "treatment" groups are created organically, by the operation of law or policy. Say that a country's UBI stipend is distributed only to individuals born after January 1, 1970. Individuals just before the eligibility date can be put into a "control" group [*2323] while individuals just after it can be put into a "treatment" group. Differences in their outcomes can be attributed to the treatment. To take an example from the patent system, patent rights are territorial, and the same patent application is often filed in multiple patent offices by applicants seeking coverage in different countries. This creates a natural experiment involving "identical twin" applications, by which the impact of different forms of examination on the same application can be compared.

"Lab" or Simulated Experiments: Another way to carry out controlled policy pilots is through "lab" or simulated experiments. To test how best to distribute a UBI, for example, one could imagine an agency contracting with a lab to run an experiment, as the Consumer Protection Financial Bureau ("CFPB") has done. 47The pilot could, for example, vary the way that the UBI is distributed, e.g., through a credit card with all the money up front, or through a regular cash disbursement. Any differences in outcomes pertaining, for example, to the well-being and employability of the participants could then be observed. 48Though doing so wouldn't generate the same information as a full-fledged field pilot, such a pilot could provide suggestive evidence of the impact of the distribution method, if any. Of relevance to the patent system, laboratory methods have been used to test the impact of changes in the standard of proof required to invalidate a patent 49as well as to test the impact on innovation of patent versus copyright-like incentives. 50The USPTO has also created pilots that applicant can opt into. 51Like lab experiments, voluntary programs simulate but differ in important ways from mandatory programs.

Experimenting with Internal Agency Procedures: "Field" experiments vary the experiences of regulated entities. One way of doing so is by experimenting with internal agency procedures. In the case of UBI, for example, one could imagine trying different ways of disbursing stipends to individuals, e.g., through a high-or low-touch process. The outcomes of individuals that receive money one way could then be compared with the outcomes of individuals that receive it another way. In a similar vein, how the Patent Office applies the resource of patent examination can and has been varied across [*2324] applications, 52with certain applications getting more scrutiny than others. Rigorously comparing the outcomes of otherwise equivalent applications can provide suggestive evidence the impact of the extra scrutiny.

Experimenting with the Laws and Rules that Apply to Private Parties: Another way to experiment is by varying the rules that apply, not to internal agency procedures, but to the behavior of private citizens. For example, some citizens in the Finnish UBI pilots were entitled to enroll in the partial basic income program while others were not. In related work, I propose a patent pilot program in which certain applicants are entitled to waive or defer the patent eligibility requirement, and some are not. 53Comparing the outcomes of "treated" applications with the outcomes of "untreated" applications can generate information about the operation of this legal standard, in view of the other standards of patentability. An agency example comes from the Office of the Comptroller of the Currency ("OCC") which, from 1999-2001 extended special lending limits to 277 community banks as part of a pilot program and compared their outcomes with companies not receiving the treatment. 54

It is worth pausing here to note that of these four classes of pilots only the last two involve the mandatory application of different procedures or rules to similarly situated entities. When a pilot is implemented as a temporary program that applies to all, equal protection claims are avoided. 55Differences in treatment that follow the decision to opt into a pilot are the result of free choice, not government discrimination, the Supreme Court has said. 56These sorts of pilots do not raise the same fairness considerations that the third and fourth categories do and are likely to be less controversial. Even if they do not vary law or policy, they can still be useful for informinglaw or policy.

The legality of pilots of the third and fourth types - which vary internal agency processes and the rules that apply to private parties, respectively - will [*2325] depend on the particulars. For the reasons described below, agency experiments that randomize internal processes rather than outward-facing rules may be subject to less stringent procedural requirements. Other aspects of a pilot's design will also have implications for its legality. For example, any number of factors - for example, that the difference in treatment does not lead to a justiciable claim or injury-in-fact, 57that the pilot does not represent "final" agency action, 58or that the pilot is mandated by Congress 59- can reduce or eliminate certain legal risks under the Constitution or APA.

However, such design choices can also limit learning. For example, while including only voluntary participants in an experiment removes a number of concerns, it can also compromise the extent to which an experimental finding is generalizable, or its "external validity" if members of the test population differ in important ways from the regulated population, or behave differently because they know they are in an experiment. 60As such it is important to understand the standards by which the most legally challenging types of rigorous pilots - pilots that randomize the rules or laws that govern the behavior of private parties - have been and are likely to be adjudicated in federal courts. Although Constitutional and APA claims will both depend in part on a pilot's "rational" or "reasoned" basis, leading at least one appellate court to conclude that "the equal protection argument can be folded into the APA argument," 61this Part separates them for the sake of completeness.

While few cases have directly considered rigorous pilots, the logic that courts have applied to randomized governmental behavior, and to experimentation and experimental evidence more generally provide windows into how courts are likely to decide challenges to such pilots. The paragraphs below consider cases involving all three types of pilots.

[*2326]

A. Constitutional Challenges to Rigorous Policy Pilots

The Equal Protection Clause forbids States from "denying to any person within its jurisdiction the equal protection of the laws," 62and enshrines the principle that like persons deserve to be treated alike. 63The Fifth Amendment prohibits the Federal government from depriving persons "of life, liberty, or property, without due process of law" 64and has been held to import an equal protection requirement equivalent to that established by the Fourteenth Amendment. 65The concepts of equal protection and due process both "stem[] from [the] American ideal of fairness." 66Rigorous pilots that impose work program requirements on some individuals eligible for welfare but not others, that put similarly situated debtors on different fee schedules, or that entitle some but not others for early release from prison at first glance might appear to fail the basic test. However, as described below, courts considering these and related experiments have reached the opposite conclusion, upholding their constitutionality.

1. The Interests Rationally Furthered by Policy Pilots

The Supreme Court has held that in areas of social or economic policy, a statutory classification that neither burdens fundamental rights nor targets a suspect class is consistent with Equal Protection as long as it "bears a rational relation to some [independent and] legitimate [legislative] end." 67Rational basis review also applies when a party lodges a due process challenge to legislative action on the basis that it deprives persons of life, liberty, or property. 68Generally speaking, pilots that do not "proceed along suspect lines" 69such as race, alienage, or gender 70should not be subject to heightened review merely based on the way they distinguish between participants and non-participants. In fact, rigorous pilots that randomly select certain individuals to receive a treatment arguably do not discriminate in the first place, but rather provide all with an equal chanceof receiving the treatment. At least one lower court has endorsed this principle, stating in its review of a random ballot voting scheme, that "there can be no denial of equal protection when all share an equal opportunity." 71

[*2327] The types of fundamental rights that are protected by due process 72are also not generally within the ken of executive agencies. Agency pilots that provide benefits to certain individuals do not "deprive" others of life, liberty, or property. Regarding an inmate's potential early release from prison through a test program, a lower court has concluded "the Petitioner does not have a constitutionally protected liberty interest in participating in the Pilot Program." 73Most policy pilots will be reviewed under the permissive, rational basis standard.

In a post- Lochner world, courts have consistently held that the judiciary cannot "second guess the legislature on the [underlying] factual assumptions or policy considerations" 74when applying rational basis review. Although legislatively-mandated policy pilots are arguably on stronger ground than agency-initiated pilots, courts have consistently applied the same, exceedingly forgiving rational basis test to a wide variety of forms of government action 75including executive agency action. 76

Why experiment? Courts generally recognize that pilots that treat similar parties differently rationally advance a variety of government interests. In the leading case on agency experimentation, Aguayo v. Richardson, the Second Circuit was asked to evaluate the constitutionality of a policy experiment approved by the Secretary of Health that imposed work requirements on a subset of individuals eligible for welfare benefits. 77In upholding the Secretary's approval, the court held the "purpose to determine whether and how improvements can be made in the welfare system ... [to be] as "legitimate' or "appropriate' as anything can be." 78

In In re Prines, 79petitioners brought an equal protection challenge to a pilot program that required them to pay debtor fees immediately in order to fund the program while those in non-pilot districts would not be charged until a year after their districts joined the pilot. Denying their claim, the Eighth [*2328] Circuit recognized the legitimate government interest in establishing a nationwide self-supporting trustee system and found that "the assessment of a ... fee to fund the trustee program ... satisfies the rational basis test." 80Using similar reasoning, the Third Circuit found a pilot program that favored certain race-track operators over others Constitutionally legitimate because it furthered the legislature's objective of promoting the horse racing industry. 81A lower court has upheld the legality of a test program on prisoners due to its purposes of vetting the feasibility of the program for reducing prison overcrowding. 82To survive rational basis review, an agency's purpose merely needs to be a legitimate one, and the purpose of testing regulatory treatments seems to meet that standard.

These cases are consistent with decisions that have endorsed the rationality of "experimental" state action outside the strict confines of a pilot. For example, in Moore v. Detroit Sch. Reform Bd., the Sixth Circuit considered a Michigan school reform law prompted by the perception of a "crisis" and "need for immediate action" on the part of the Michigan legislature. 83Upholding the law, about which there was considerable uncertainty, the Sixth Circuit stated, "state legislatures need the freedom to experiment with different techniques to advance public education and this need to experiment alone satisfies the rational basis test." 84The Pennsylvania Supreme Court applied similar reasoning to a narrowly targeted law aimed at the state's failing schools that, due to the large number of conditions placed on it, only applied to a single district, and put the law at risk of invalidation under a prohibition on "special legislation." 85Upholding the law, the court held that the policy in question should be "assessed under the act's "pilot program' before being made more generally available ... there is nothing improper of this method of attacking social problems of statewide dimension, as the Legislature is free, for reason of necessity or otherwise, to address such issues incrementally." 86

2. Unequal or Equal Treatment?

Do controlled experiments raise particular due process or equal protection concerns? In Aguayo, described above, the contested work program [*2329] requirements applied to a fraction of the total eligible population. 87In upholding the program, the Second Circuit held that the government interest in finding what worked to be ""suitably furthered' by a controlled experiment, a method long used in medical science which has its application in the social sciences as well." 88

Courts have also generally rebuffed constitutional challenges to government actions based on their random distribution of burdens or benefits. In Engquist v. Or. Dep't of Agric., the Supreme Court considered an Equal Protection challenge brought by a public employee who alleged that she was fired arbitrarily, while other similarly situated employees were not. In denying her "class-of-one" claim, the Court stated that "a random choice among rational alternatives does not violate the Equal Protection Clause." 89A number of circuits have similarly held that a violation of equal protection requires discrimination that is intentional and not merely the product of "random government incompetence." 90

These endorsements of intentional randomness complement court acceptance of randomness that arises out of necessity, for example when demand outstrips supply. 91When a benefit or regulation is distributed to a limited group as part of a pilot, randomization in allocation prevents unfairness, giving all stakeholders an equal chance of receiving the treatment even if they don't all do so. However, while randomness is therefore tolerated, and even at times has been encouraged, 92by the courts, it is not required. 93

B. Challenges to Experimental Rulemaking

Federal agencies must also worry about challenges to their actions under the APA. Whether or not an agency uses experimentation, its processes must be procedurally adequate and withstand judicial review. 94As described below, courts will consider both the form (e.g., does the pilot involve a procedural [*2330] or interpretive rule) and the consequences of a policy pilot when reviewing the relevant administrative record. While conducting a policy pilot or forming a final rule based on an experiment can raise distinct issues, each experimental action of an agency, just like each non-experimental action of an agency, must be evaluated on its own merits.

1. Administering Policy Pilots

Agency statements "designed to implement, interpret, or prescribe law or policy or describing the organization, procedure, or practice requirements of an agency" are considered rules governed by the APA. 95The APA's informal rulemaking procedures require agencies to provide the public with adequate notice of a proposed rule and enable interested persons a reasonable and meaningful opportunity to participate in the rulemaking process. 96Announcements of adopted rules must be accompanied by "a concise general statement of their basis and purpose," 97and the agency's responses to "significant" comments. 98

Whether or not a pilot is exempt from notice and comment requires a case by case determination, 99and agencies have exercised discretion accordingly. However, pilots that "have considerable impact on ultimate agency decisions" or "substantially affect[] the rights of those over whom the agency exercises authority," 100even those that merely vary agency procedure, 101require notice-and-comment. Adopting a rule on an experimental, then permanent basis can require the agency to go through an additional round or rounds of public comment. 102But in light of the high failure rate of experiments, 103it will often be the case that a practice is abandoned after a pilot has run.

While a voluntary, limited or temporary pilot will have less impact than a mandatory, agency-wide, or permanent practice, there's also a risk that the [*2331] uncertainty associated with a temporary program will unsettle stakeholders and result in more speculative and premature reactions than would be elicited by a more formed proposal. 104To reduce these risks, an agency's disclosure should at least inform the public of: the motivation for the pilot, the policy to be tested out, the information sought, and (where available) the agency's commitment to action based on the results of the pilot - a best practice in experimentation called "pre-commitment." Rulemaking based on experimental evidence may require more disclosure. The Portland Cement 105decision of the D.C. Circuit and its progeny have been read to suggest "an obligation to reveal agency data from the outset," 106in order to support review and engagement with the data when dispositive. 107In cases where the data is conclusive, even if more work to disclose, with the shared experience comes a greater chance of a common, ground truth, at least that is the hope. 108As long as the agency faithfully follows the ordinary notice and comment process, procedural obstacles should be surmountable.

2. Judicial Review of Experimental Agency Action

The APA requires courts to set aside reviewable agency action that is "arbitrary, capricious, an abuse of discretion, or otherwise not in accordance with law" 109or which otherwise exceeds the agency's constitutional or statutory authority. 110Under State Farm, this requires the agency to "examine the relevant data and articulate a satisfactory explanation for [the] action including a "rational connection between the facts found and the choice made.'" 111Courts are not permitted to substitute the agency's judgment with their own, 112but rather are required to ensure that the agency's explanations are satisfactory and do not run counter to the evidence. 113

When an agency is implementing a policy pilot pursuant to a statutory authorization, judicial review will necessarily involve consideration of Congress' objectives and intents to ensure that an agency has not

[*2332]

relied on factors which Congress has not intended it to consider, entirely failed to consider an important aspect of the problem, [or] offered an explanation for its decision that runs counter to the evidence before the agency, or is so implausible that it could not be ascribed to a difference in view or the product of agency expertise. 114

While State Farm applies to agency action generally, not just agency piloting, the reversible nature of a pilot may bear upon the court's factual evaluation and cause a court to somewhat soften its hard look, at least, relative to a permanent irreversible version of the policy.

For example, in the Aguayo case described earlier, the Secretary's approval of a pilot program was made pursuant to 42 U.S.C. § 1315, which authorizes statutory waivers to support welfare demonstration projects. In reaching its decision, the Second Circuit considered the statute's objectives, the Secretary's consideration of the contested programs in view of the objectives and evidence, and the adequacy of the administrative record, including the agency's consideration and response to objections raised by the appellant. 115But the court also relied upon "the fact that the programs were of limited duration and would remain under ... on-going supervision" 116to endorse the setting of "a lower threshold for persuasion" for the approval of experimental, temporary programs. 117Finding there to be "no "clear error of judgment,'" 118the court specifically noted that "experience will be the best test of the reality of appellants' fears." 119

While this single case by no means endorses a general heightened level of deference for policy pilots, it does underscore the fact-intensive nature of judicial review and the willingness of courts to engage in experimental agency action on its particular merits.

3. Judicial Deference or Indifference to Experimental Evidence?

What about agency rules that rely upon experimental evidence? 120The D.C. Circuit has articulated some level of heightened deference to experimental evidence. In a 2002 decision involving a two-year experimental waiver on capacity ceilings by the Federal Energy Regulatory Commission, the Circuit began its analysis by citing "the special deference due agency experiments" and reiterating the Commission's statement, that, "no matter how good the data suggesting that a regulatory change should be made, there [*2333] is no substitute for reviewing the actual results of a regulatory action." 121The court went on to describe its practice of, "for at least 30 years ... [giving] special deference to agency development of such experiments, precisely because of the advantages of data developed in the real world." 122

However, the arguably more defensible way of regarding experimental data is to treat it like any other form of evidence - on its own merits, in its own individual context. Rigorous experimental data canand often will besuperior to other forms of data and deserve more deference. This appeared to be the case in Christ the King Manor, Inc. v. Sec'y U.S. Dep't of Health & Human Servs., in which the Third Circuit had occasion to evaluate the Secretary of Health and Human Services' reliance on "actual-experience data" that showed the impact of a proposed rate change on the quality of Medicaid care. 123Because the experiential data was "clearly more recent and more accurate than the [older] predictive data," the court found, it made little sense for the Secretary to rely on the older data and "ignore the "answers to the test.'" 124But there will be situations in which experimental data will notdeserve special deference or elevation above other sources of data. As discussed in the next Section, any number of defects including methodological issues, unobservable factors and outcomes, changes in conditions, or conflicting or inconclusive results can limit what can be learned from a pilot, however experimental.

An indifferent, rather than deferential, posture is supported by a number of decisions that analytically separate State Farminto two distinct inquiries, one far more important than the other. 125The first is to ascertain the data considered by the agency for its relevance, and the second, critical step, is to consider whether or not the agency's choice or action rationally follows from them. The way that courts have regarded non-experimental evidence associated with the first step - as imperfect, fallible, and in most cases, adequate - suggests they would apply a similar approach to experimental evidence. For example, several appellate courts have held that the consideration of older 126or second-best 127data by itself, is not arbitrary and [*2334] capricious. 128In a similar vein, the D.C. Circuit has ruled that anecdotal observations can suffice, as long as the inferences drawn from them are reasonable. In the case of Nat'l Black Media Coal. v. FCC, for example, it blessed "the Commission's decision to forego a rigorous statistical evaluation," in view of the modest inferences it made from its analysis, to be reasonable. 129There is no affirmative obligation to experiment. 130

But what courts have been unwilling to tolerate are misuses of data to reach unsupported conclusions. Selectively relying on portions of a study while ignoring other, contrary parts of the study has been held to be unreasonable. 131So has willfully ignoring new and better data within an agency's grasp, 132though whether data are "better" is often closer to contestable opinion than objective fact. In addition, while an agency's embrace of experimentation can demonstrate that the agency is considering the effect of its policy, it does not necessarily follow that the subsequent policy developed by the agency is reasonable, the Ninth Circuit has found. 133

C. Conclusion

Collectively, these cases confirm that the Constitution and APA present limited if any special barriers to rigorous piloting. No decision that I could find has held that experimentation rendered an agency rule per se arbitrary and capricious because it demonstrated a lack of sufficient information to select a course of action. To the contrary, courts are generally accepting of experimentation as a rational way of learning about a policy and value [*2335] randomization as an adequate, and potentially superior, 134way of distributing a benefit or a burden to a population.

At the same time, as described above, rigorous experimentation will neither remove the burdens of ensuring an adequate nexus to a legitimate governmental objective, nor absolve an agency of procedural compliance with notice and comment, nor shield agency from scrutiny. When the stakes are high and agency action is likely to have material consequences for stakeholders, experimental evidence, like other evidence, is likely to be contested and challenged. While on one hand, experimental evidence can create a shared experience, on the other hand, stakeholders and even agencies are likely to interpret evidence in the manner that advances their interests. 135Matters of institutional competence may come into play if the court is unsure how to evaluate an an agency's experimental design or interpretation of experimental results.

Still, for satisfying an agency's own learning agenda, and injecting greater certainty into its decisions, "hard" experimental evidence will often compare favorably. While feedback through litigation requires the courts to address petitioner's question, experimentation allows the policy-or law-maker to ask her question, as well as specify the method that will be used to answer it. Expert opinion evidence provided through lobbying and participation in notice-and-comment will in some cases be more partial and less representative than experimental evidence. But even if legal, in what situations is experimentation going to be practically feasible and desirable? The answers that agencies have given to this question forms the basis for the next Part.

III.

(A Framework For Proposing) Rigorous Policy Pilots (That) Are Feasible and Worthwhile

The previous Part addresses legal challenges to rigorous policy pilots. However, significant practical and institutional challenges to rigorous piloting will often attend. Recent research has found that people tend to view as inappropriate pilots that purport to randomize a treatment even when the underlying treatment is unobjectionable, 136suggesting that buy-in of the [*2336] relevant population, if required, will require education and communication. More upfront work is also required to make explicit the assumptions behind an intervention, and then to develop a strategy for testing them. Agencies carrying out rigorous tests must marshal resources, data, statistical expertise, support for departing from the status quo, and explanations for skeptics. 137There often will be no agreed upon measure of success. Even when there is, attrition, changed circumstances, poor design or a host of other factors can lead to a lack of inconclusive or null results. The results may be rejected. For example, in their review of the 50 or so randomized experiments in U.S. law that have ever been conducted, mostly in the field of criminal law and judicial procedure, Greiner and Matthews found a major problem to be the refusal of judges and lawyers to accept experimental results that did not jive with their sense of the "right answer." 138

However, as daunting as the hurdles to rigorous piloting may appear, they are not new. Although individually perhaps little known beyond each relevant agency or initiative and attendant stakeholder community, collectively, the experience of executive agencies with rigorous pilots is not insubstantial. Outside of medical contexts, 139randomized controlled trials ("RCTs") were carried out in waves during the 1930s-1970s to address social policy questions, and then again in the 1990's to support the development and implementation of welfare and employment laws. 140Between 2009 and 2011, Congress enacted six "tiered-evidence" programs at multiple agencies to fund replication RCTs to determine whether prior findings could be reproduced on a larger scale. 141The Institute of Educational Sciences within the Department of Education, 142Center for Medicare and Medicaid [*2337] Innovation within the Centers for Medicare & Medicaid Services, 143and Office of the Chief Evaluation Officer ("CEO") within the Department of Labor ("DOL"), 144each support rigorous evaluation. 145Various Presidents have also called for rigorous controlled testing of policies. 146

It is also worth remembering that calls for rigorous evaluation are not new and their results have not necessarily been encouraging. Critics have described evaluation mandates embedded in the Government Performance and Results Act of 1993 ("GPRA"), the GPRA Modernization Act of 2010 ("GPRAMA"), and successive generations of retrospective review requirements as hampered by resource constraints, pro forma compliance, and a lack of oversight. 147

[*2338] However, several developments make it an opportune time to persevere and attempt to answer, not abandon their call. First the costs of controlled experimentation are going down, thanks to the advance of open data, 148and the adoption of "agile" and "policy lab" approaches in government, 149which have increased the expertise and decreased measurement and assessment costs. Second, the embrace of performance-or results-based lawmaking has shifted attention away from policy processes and towards policy outcomes, discoverable through trial and error or disciplined experimentation. 150Third, the 2018 Evidence Act, if implemented as intended, should significantly shore up agency capacity for rigorous evaluation. The Act requires departments and agencies within the Act's jurisdiction to submit annual plans for identifying and addressing policy relevant questions that specify "data to ... facilitate the use of evidence in policy making." 151It establishes an Interagency Council on Evaluation Policy, requires each relevant agency to designate a Chief Evaluation Officer and requires the Office of Management and Budget ("OMB") to establish an Advisory Committee on Data for Evidence Building to advise on expanding access to and use of federal data for evidence building. 152These activities should inform and ease at least two of the steps of rigorous piloting described below - identifying important questions and developing the evidence to evaluate them.

[*2339] Finally, many agencies are already actively engaging in piloting. 153While rigorous designs are by no means part of the standard repertoire, 154designing policies with identification strategies in mind can improve the chances that the actual impact of law or policy on outcomes will be discernible.

Can practical obstacles to rigorous piloting be overcome and when is it worthwhile to do so? The next Sections draw on the past experiences of agencies, including the USPTO, to illustrate that rigorous pilots are not only feasible but can be informative and useful. They present a framework for proposing pilots that comprises specifying (1) questions that matter; (2) authority and agency resources; (3) theory of change; (4) testing protocol; (5) evidence and (6) resources ("MATTER").

A. Step 1 ("M"): Select a Question that Matters

The types of questions that rigorous policy pilots have addressed in the past provide some insight into the questions that could be feasible and worthwhile to be addressed in the future. They span diagnostic questions (how accurate is the Trademark registry?, 155asked by the USPTO), policy development questions (what is the impact of price-test restrictions on short [*2340] sales?, 156asked by the SEC), long-term outcomes questions (do housing and services interventions for homeless families prevent them from returning to homelessness?, 157asked by HUD), and policy implementation questions (what strategies work best for driving benefits uptake?, asked by multiple agencies 158). It is hard to imagine other ways of getting evidence of comparable quality about the performance (as opposed to, say, the popularity) of legal and policy interventions.

But as wide-ranging as the pool of potentially good questions might be, the range of questions poorly suited for rigorous experimentation is also considerable. They include situations where priors, dictated for example by politics or ideology, are so strong that even high-quality evidence is unlikely to be persuasive, as well as situations where the importance of implementation and contextual factors make it difficult to replicate observed effects. 159Situations where causal proof would be frivolous, 160for which the important outcomes are not subject to measurement, 161or where the observations would be too few perhaps because the questions are so big, encompassing imagining and measuring the impact of major changes in institutions, for example, also present poor candidates.

These criteria, in turn, make rigorous experimentation easier to carry out with respect to some agency actions than others. Agency interventions that are customized and non-repeatable, for example involving long-term investigations, are not easily tested through field RCTs. However, lab tests that vet mechanisms, address basic research questions, or simulate market dynamics can still be useful. The CFPB, which is tasked with taking enforcement actions, has used rigorous testing to increase its understanding of market mechanisms, for example, by lab-testing disclosures and simulating consumer environments and markets to explore how the complexity of a product's price influences the consumption decision. 162

Do these constraints shrink the pool of questions addressable by rigorous pilots to the uncontroversial, transactional, and therefore marginal? Over [*2341] several decades, 163there has been a debate about the shortcomings of insisting on causal methods and their tendency to favor, "good answers instead of good questions." 164As Raj Chetty has commented, "people think about the question less than the method ... so you get weird papers, like sanitation facilities in Native American reservations." 165But keeping at the center of the inquiry the informational deficits that only pilots can address, and how pilots fit into a broader learning and law-making agenda can ensure that experimentation is worth the candle.

For example, when the HHS started advancing its controlled welfare experiments for evaluating welfare policy, it focused on "the most significant questions about policy and practice," 166to get buy-in from relevant administrators and agency employees responsible for implementing the policy. The RCTs "did not assess reforms dreamed up by policy wonks ... [but rather] bubbled up from governors, their staffs, and community activists - people with finely calibrated judgment on political timing." 167Even though the resulting programs took years to design and implement, the work remained relevant to a wide variety of stakeholders. During the 1980's and 1990's, HHS funded or facilitated more than 85 welfare RCTs, "building a sizeable body of credible evidence with clear policy relevance." 168Among the findings: Short-term job-search assistance and training had larger effects than remedial education and combining mandatory participation in employment-focused services with earning supplements was also effective at raising overall income and moving individuals out of poverty. 169The results informed broader welfare lawmaking and policy development in the United States. Unemployment insurance ("UI") policy is another context in which rigorous pilots have been used to answer open question in law-and policy-making. 170

As such, among policy actors, agencies enjoy certain comparative advantages (though not free reign), to generate information. Unlike courts, which are constrained by the parties and facts before them, pilot designers [*2342] are free, within the limits of their authority, existing law, and related considerations, to design and adapt suggested policy interventions. Picking questions that are relevant to all branches of government, but which exploit an agency's comparative advantages can be particularly generative. Pilots can also be used to vet competing claims from stakeholders. For example, in 2018, the USPTO proposed a new procedure for amending patents challenged in post-grant proceedings. One aspect of the new procedures that was highly contested among the 49 responses it received concerned the appropriate timeline for amendment. Rather than going back to the drawing board, the USPTO noticed a pilot version of the program, and noted that it reserved the right to amend its terms in response to feedback and experience generated during the pilot. 171

B. Step 2 ("A"): Consider Existing Authority and Agency Resources

As discussed in Part II, the legal requirements for carrying out pilots related to an agency's objectives need not be particularly strenuous. But marshalling the necessary agency approvals can present major hurdles. Implementing pilots can require employees to vary their behavior or depart from the normal course of operations, introducing both time and disruption costs.

Pre-decisional consultation processes can allow labor and agency to reach agreement about the terms and conditions of a pilot. 172For example, when the USPTO decided to pursue a pilot that would require unionized examiners to add more detail to the patent record, agency leadership and labor negotiated and conferred. The resulting memorandum of understanding that detailed eligibility criteria, how examiners would be invited to participate in the process, and how data would be shared by both labor and management to evaluate the pilot's success. 173

Another way to justify rigorous pilots is to integrate them into existing compliance obligations. The Administrative Conference of the United States has recommended using rigorous evaluations to fulfill various retrospective review mandates using a framework that closely resembles the one discussed in this Essay. 174When rigorous retrospective evaluations take place, the results [*2343] are often surprising: The Drug Resistance Education programs ("D.A.R.E") of the 1990s and 2000s, implemented in more than 75% of U.S. schools, 175wrap-around services for prisoners reentering society, 176deworming, 177and 90% of the interventions evaluated by the Institute for Education Sciences from 2002 to 2013 178are among the interventions that have been found to be ineffective.

C. Step 3 ("T"): Identify a Treatment and Theory of Change

While pilots can be used to evaluate already-implemented policies, prospectively proposing a policy pilot involves identifying an intervention for evaluation grounded in a theory of change. Evidence is likely to be most valuable and valued when it addresses deficits left by traditional information gathering processes. Tests can address competing stakeholder narratives and theories of change for example, or can be used to try out new ideas. 179Within the USPTO, participants in the Office's "Edison Scholar" program have helped to develop pilot programs including the "Glossary pilot," 180other pilots have come from or been supported by the Public Patent Advisory Committee ("PPAC"), 181a Federal Advisory Committee of the agency.

Regardless of source, it is critical to connect the identified treatment to the eventual intended outcome. By answering the question, "how is treatment X supposed to achieve outcome Y," a law-or policy-maker describes a "theory of change." A widely-accepted method of specifically testing the underlying theory of change is through "mechanism experiments," a technique for [*2344] determining whether the underlying theory of change is actually sound. 182Mechanism experiments support testing of the core underlying assumptions without requiring an extensive randomized controlled experiment; 183as described in Part II, to the extent they simulate rather than actually effect policy changes, they impose fewer administrative burdens.

D. Step 4 ("T"): Specifying the Test Strategy

Rigorous piloting involves selecting a test strategy that is appropriate to the question being asked. A "mixed methods" strategy of getting feedback, e.g., through surveys, interviews, case studies, 184and statistical (causal and non-causal) approaches is often best. Regardless of the specific method of evaluation, though, each of the other parts of specifying a pilot: articulating the question to be addressed, a theory of change, the evidence, how to measure it, and matching resources to the pilot, will require attention and in some cases, consultation with stakeholders, cost and convenience trade-offs, and implementation risk. In some cases, developing the right metric will be transformative, in others, framing the question correctly can unlock greater learning.

But when the question is, "did policy X have Y effect," experimental or quasi-experimental approaches are understood to yield the best information. Volumes have been written about appropriate methods of evaluations. 185But also worth underscoring is what government pilots have revealed about the value of rigor, and the wasted opportunity associated with the lack thereof.

Section 1115 of the Social Security Act allows states that receive approvals from HHS to test and evaluate new approaches for delivering Medicaid services using funds that would not otherwise be eligible. 186Taking advantage of these waivers, by 2016, nearly three-quarters of states implemented at least part of their programs under demonstrations. 187To evaluate what had been learned, the Government Accountability Office ("GAO") reviewed demonstration projects in eight out of 15 states. The results were not good. The GAO concluded that a lack of care in how the evaluations were carried [*2345] out, reported, and shared compromised their usefulness. 188The lack of control groups in four out of the eight states, insufficient sample sizes and response rates for surveys, and the inability to isolate the impact of the Medicaid interventions from other changes limited what could be concluded about the effectiveness of demonstrations. 189Rather than resource deficits being the problem - in 2015 alone, states spent over $ 100B, or one-third of Medicaid program expenditures on "demonstrations" - the problem seems to have been systemic attention deficits, to specifying upfront how evaluations could meaningfully be done, 190for example by implementing them in a way that would support experimental or quasi-experimental evaluation, as well as, perhaps a lack of a bona fide intent to test. These "significant limitations ... affected their usefulness in informing policy decisions." 191

This description suggests that the lack of close alignment between those requesting and those implementing pilots can drastically reduce their value as a learning exercise. 192One way to avoid this result is by ensuring that interventions are grounded, or at least framed, in terms of the lived experiences of those required to implement them. Presenting "rigorous pilots" as tools for advancing and testing assumptions and theories developed from the "bottom up" can distinguish them from "top-down" evaluation mandates that are not well integrated into agency decision-making.

Sign in to recut, update, or save this card.