Search the archive.

A wage subsidy has the potential to both increase the earnings of low-skilled workers and expand the job opportunities available to them, helping to reduce poverty, increase labor-force participation, and boost economic growth. The ongoing economic crisis in Puerto Rico offers an ideal testing ground for the policy, and the policy offers an ideal tool for addressing the island’s woes.

At the root of Puerto Rico’s crisis lies a dysfunctional labor market. Only 40 percent of adults are employed or looking for work, thanks largely to a federal minimum wage misaligned with the island’s local market conditions. The federal minimum of $7.25 per hour has a relatively small effect on the mainland, where the median hourly wage is more than $17.40 and only 3% of workers earn less than $8.50. In Puerto Rico, by contrast, the median hourly wage is $9.61 and 28% of workers earn less than $8.50.

Policy pilot evaluations often adopt a relatively simplistic approach of considering whether or not a pilot has “worked” against particular outcome criteria. We have shown that this represents a limited understanding of the purposes and effects of pilots. We have brought together literature on policy success with that on policy pilots to generate a framework within which empirical evaluation findings can be synthesized with analysis of the wider policy landscape to consider pilot success in a more nuanced and multifaceted way. While no such framework can be completely comprehensive, and judgments as to “success” in each category will be contingent, provisional, and potentially arguable, we would suggest that the most valuable aspect of the framework is its focus on explicit delineation of different dimensions of success and on the declaration of the perspective being adopted. While different commentators may disagree with particular judgments in each cell of our summary table, the criteria by which we are judging and the specific aspect of the pilot being judged are clear, providing a more nuanced evaluation framework and facilitating constructive discussion.

Furthermore, we believe that explicitly considering pilots across all of these dimensions holds promise in supporting the design of more comprehensive and nuanced evaluation programs which move beyond a simplistic attempt to demonstrate “what works.” We have also shown how the explicit separation between program and political elements of success allows interrogation of the antecedents of particular policy decisions, facilitating deeper understanding of contemporary policy trajectories. This in turn supports a more nuanced understanding of later policy developments. For example, we suggest that early political claims to the success of the Vanguard program may have inhibited longer-term consideration of the practical steps required to facilitate program success, with potentially significant implications for NHS managers seeking to implement new approaches to service integration. Moreover, we would suggest that our framework facilitates a more nuanced understanding of pilots, looking beyond stated rationales, and encouraging evaluators and others to explicitly consider the extent to which particular pilots do indeed represent a rational attempt to test policy ideas, or whether they in fact represent an approach to implementation. “Success” can then be judged against these more complex objectives. We invite others interested in this area of research to consider the applicability of this approach in other fields, with a particular focus upon issues of temporality and methods to identify criteria by which political success might be judged.

One way to address both the knowledge gap and risk aversion in policy development is by introducing a temporary change to law or policy for the purpose of learning from it, through a “policy pilot.” Implemented with rigor, through the application of, “well-designed and well-implemented methods tailored to the questions being asked”13—rigorous policy pilots are a generative yet under-used tool for addressing informational deficits that stand in the way of developing effective law and policy.

The range of open questions in law and policy that can be addressed by pilots are wide-ranging. They include, for example, the question of how giving a universal basic income (“UBI”) to individuals impacts their well-being and employment. In 2017, the Finnish government began tracking the outcomes of 7,000 unemployed citizens, about a third to which it gave $600 a monthto address this question;14 several U.S. jurisdictions are planning their own tests.15 Also unknown is how the administration of patent law and issuance of quality patents best advance the Constitutional goal of “promot[ing] the progress of science and useful arts.”16 Over the last decade, the United States Patent and Trademark Office (“USPTO”) has run numerous pilots to address this question.17 As the D.C. Circuit has said, “there are some situations in which, ‘a month of experience will be worth a year of hearings.’”18 While experiments or pilots can address a variety of questions pertaining, e.g., to a policy’s feasibility or stakeholder reactions, when the question concerns whether the policy has caused an observed outcome, a controlled trial that provides the treatment to one part of population, withholds it from another, and then compares the difference in outcomes provides the best insight. Such rigorous experiments can be designed, in turn, with or without randomization, and to vary the rules or laws that apply or to approximate such changes. A pilot that limits to whom a rule or policy applies constrains the risk but not the learning. As Justice Brandeis famously dissented, “a single courageous state may . . . serve as a laboratory; and try novel social and economic experiments without risk to the rest of the country.”19

On the other hand, Andrew Yang, who is running for the 2020 Democratic presidential nomination in the US, says the experiments that have already been run are enough to prove that basic income can be successful. “It’s wrong to think we don’t have information on this. We do,” Yang told MIT Technology Review. “There have been many implementations of basic-income-aligned programs over the last number of decades.”

He’s right in that this is not a new idea: various programs have been around for a while, including the Alaska Permanent Dividend Fund, which has been providing data on basic income since 1982. An earlier incarnation of the idea was supported by the conservative economist Milton Friedman in the 1960s, and both Richard Nixon and his Democratic opponent, George McGovern, supported some version of it during the 1972 campaign. It was a rare instance of bipartisan support.

Interest in universal basic income has grown in recent years, and now many are looking to pilot programs to see whether local-level experiments could be applied to the federal level. The concept of UBI, which is paying a person merely to exist and not in exchange for labor, has been around for decades. But interest in the idea grew amid presidential candidate Andrew Yang’s push for UBI as well as the rounds of stimulus spending that the federal government churned out during the coronavirus pandemic.

There have been several pilot programs for UBI, although it is worth noting that opponents of the concept are quick to call those experiments flawed for many reasons. Critics contend that there has never been a scientifically sound UBI pilot because of how the local programs are structured — and that UBI on a broader level would be even more difficult. Proponents of UBI argue that a broad-based program of direct payments could lift families out of poverty and even make employment more tenable because of fewer economic barriers. They also see it as a way that families could overcome healthcare and education challenges. But critics say that any sort of national UBI program would be prohibitively expensive and could end up hurting more than helping due to the massive tax burden involved.

Natural Experiments: In some cases, “control” and “treatment” groups are created organically, by the operation of law or policy. Say that a country’s UBI stipend is distributed only to individuals born after January 1, 1970. Individuals just before the eligibility date can be put into a “control” group while individuals just after it can be put into a “treatment” group. Differences in their outcomes can be attributed to the treatment. To take an example from the patent system, patent rights are territorial, and the same patent application is often filed in multiple patent offices by applicants seeking coverage in different countries. This creates a natural experiment involving “identical twin” applications, by which the impact of different forms of examination on the same application can be compared. “Lab” or Simulated Experiments: Another way to carry out controlled policy pilots is through “lab” or simulated experiments. To test how best to distribute a UBI, for example, one could imagine an agency contracting with a lab to run an experiment, as the Consumer Protection Financial Bureau (“CFPB”) has done.47 The pilot could, for example, vary the way that the UBI is distributed, e.g., through a credit card with all the money up front, or through a regular cash disbursement. Any differences in outcomes pertaining, for example, to the well-being and employability of the participants could then be observed.48 Though doing so wouldn’t generate the same information as a full-fledged field pilot, such a pilot could provide suggestive evidence of the impact of the distribution method, if any. Of relevance to the patent system, laboratory methods have been used to test the impact of changes in the standard of proof required to invalidate a patent49 as well as to test the impact on innovation of patent versus copyright-like incentives.50 The USPTO has also created pilots that applicant can opt into.51 Like lab experiments, voluntary programs simulate but differ in important ways from mandatory programs

Experimenting with Internal Agency Procedures: "Field" experiments vary the experiences of regulated entities. One way of doing so is by experimenting with internal agency procedures. In the case of UBI, for example, one could imagine trying different ways of disbursing stipends to individuals, e.g., through a high-or low-touch process. The outcomes of individuals that receive money one way could then be compared with the outcomes of individuals that receive it another way. In a similar vein, how the Patent Office applies the resource of patent examination can and has been varied across applications, with certain applications getting more scrutiny than others. Rigorously comparing the outcomes of otherwise equivalent applications can provide suggestive evidence the impact of the extra scrutiny.

The pilot project is a specific work method, fulfilling a function of experimentation, which is mobilised in a plethora of sectors and allows a learning while doing approach (Kay & Boxall, 2015; Lee, 1999). This tool offers the possibility to test the innovation at stake in real-world settings at small and controlled scales, which limits the impact of potential failure or negative side effects (Vreugdenhil & Ker Rault, 2010). Furthermore, the actions undertaken as part of the experiment are expected to be more reversible (Musselin, 2005; Zurlo & Nunes, 2016) and, for this reason, are supposed to be less frightening for the stakeholders involved.

3Therefore, pilot projects are often used by policy makers in innovative contexts (Vreugdenhil et al., 2010), to learn from the experimentation carried out (Engström & Lidelöw, 2015) and to create new knowledge and produce evidence for policy-making (Sanderson, 2002; Vreugdenhil & Ker Rault, 2010). They are thus expected to help orient future political action and policy development. In this way, the pilot project constitutes a tool for, “improving the effectiveness of policy responses” (Sanderson, 2002, p. 4). The purpose is to work collaboratively (Zurlo & Nunes, 2016) in order to identify what works in what context (Sanderson, 2002; Vreugdenhil & Ker Rault, 2010).

The Federal Jobs Guarantee Development Act, announced by Booker on Friday, would establish a three-year pilot program in which the Department of Labor would select up to 15 local areas (defined in the bill as any political subdivision of a state, like a city or a county, or a group of cities and counties) and offer that area funding so that every adult living there is guaranteed a job paying at least $15 an hour (or the prevailing wage for the job in question, whichever’s higher) and offering paid family/sick leave and health benefits.

The idea of a government job guarantee ensuring all adults who want employment get it has a long history in American politics, but it has gained popularity as the Democratic Party has sought to embrace bigger and more ambitious economic policies in the wake of the 2016 election.

(Dylan, “Cory Booker’s new big idea: guaranteeing jobs for everyone who wants one,” pg online @ https://www.vox.com/policy-and-politics/2018/4/20/17260578/cory-booker-job-guarantee-bill-full-employment-darity-hamilton //um-ef)

Sen. Cory Booker (D-NJ) has a big idea: give 15 local areas federal money so they can guarantee all their residents a job. The Federal Jobs Guarantee Development Act, announced by Booker on Friday, would establish a three-year pilot program in which the Department of Labor would select up to 15 local areas (defined in the bill as any political subdivision of a state, like a city or a county, or a group of cities and counties) and offer that area funding so that every adult living there is guaranteed a job paying at least $15 an hour (or the prevailing wage for the job in question, whichever’s higher) and offering paid family/sick leave and health benefits. The idea of a government job guarantee ensuring all adults who want employment get it has a long history in American politics, but it has gained popularity as the Democratic Party has sought to embrace bigger and more ambitious economic policies in the wake of the 2016 election. Booker’s plan is essentially a pilot version of a proposal from Duke’s Mark Paul and William Darity, Jr. and the New School’s Darrick Hamilton, economists who have, together and individually, advocated a job guarantee for years, well before the current surge in enthusiasm. Most recently, Paul, Darity, and Hamilton wrote up their proposal for the Center on Budget and Policy Priorities, a highly influential left-of-center think tank, in a clear sign the idea was gaining traction. Booker’s bill is an even bigger step forward. “The federal jobs guarantee is an idea that demands to be taken seriously,” Booker said in a statement. “Creating an employment guarantee would give all Americans a shot at a day’s work and, by introducing competition into the labor market, raise wages and improve benefits for all workers.” The policy case for a job guarantee The idea of a job guarantee serves both a policy and a political purpose. On the policy side, a job guarantee would, in theory, effectively end recessions in America. Right now, the US government’s strategy when the economy stops growing is to use a combination of monetary stimulus (in which the Federal Reserve cuts interest rates or buys up billions of dollars of long-term bonds) and fiscal stimulus (as in the 2009 stimulus package, which blended a boost in spending with temporary tax cuts). For a variety of political and institutional factors, the Fed and Congress weren’t able to do enough in 2008 to 2010 to prevent unemployment from breaking 10 percent, and they certainly weren’t able to effect a full recovery within a couple years. Returning to normal unemployment rates took nearly a decade, resulting in years of human misery and lost wage gains that a healthier and faster recovery could’ve delivered. Job guarantee advocates argue that their plan effectively creates a permanent form of fiscal stimulus that politicians wouldn’t need to scramble to pass whenever disaster hits. Instead, if the economy took a turn for the worse and companies started shedding jobs, the government would automatically soak up anyone who’s laid off and give them work. That, in turn, would put more money in consumers’ pockets, boosting demand and improving business’ prospects. Before you know it, the economy’s back to normal. Advocates argue the policy could prove beneficial during economic booms as well. The fruits of economic recoveries and booms aren’t evenly shared, demographically. As of March 2018, 60.7 percent of white people in America were employed, but only 58.4 percent of black people were. Once black men’s disproportionate representation in prisons and jails is accounted for, the gap grows still larger. In 2016, the most recent year for which we have data, a mere 27.7 percent of people with disabilities age 16 to 64 were employed, compared to 72.8 percent of nondisabled people. Nor do booms affect all geographic areas equally. There are still 337 counties or county-equivalents with a combined labor force of over 6.7 million people that have unemployment rates of 7 percent or higher as of February 2018. Seventy-eight counties, like Yuma, Arizona, still have unemployment rates in excess of 10 percent. A job guarantee would, in principle, lessen those inequalities. People in struggling regions would be guaranteed work without having to move. Black Americans and disabled Americans wouldn’t be expected to wait until employers have run out of white and nondisabled people to hire until they get their chance. Finally, advocates argue that the policy would lead to an increase in wages for everyone. That’s a big benefit at a time when unemployment is low but wage growth is still sluggish. Think of it this way: Walmart pays its employees a minimum of $10 per hour; part-time employees aren’t guaranteed benefits like health insurance or a 401(k) match. If you’re a part-time employee at Walmart, and all of a sudden you can get $15 an hour, work full time, and earn full benefits by working for the federal government — wouldn’t you? And, knowing that, wouldn’t Walmart try to increase wages to keep you? Advocates say Walmart would. And they have some empirical evidence on their side from India, where a type of job guarantee known as the National Rural Employment Guarantee Scheme functions largely as an insurance system, offering a source of income for rural farmers during the dry season. A group of economists — UC San Diego’s Karthik Muralidharan and Paul Niehaus and the University of Virginia’s Sandip Sukhtankar — found that the job guarantee, they estimate, increased earnings for low-income households by 13.3 percent. Ninety percent of that increase is due to higher wages and increased work in the private sector, not the job guarantee program itself. Just as job guarantee advocates would predict, the program bid up wages everywhere. Perhaps the most surprising result was that the program not only increased wages, but increased employment in the private sector. The political case for a job guarantee — and the challenges it faces The fact that Cory Booker, a rising star in the Democratic Party and a likely 2020 contender who faces considerable skepticism from the left due to his stances on education and Wall Street, has endorsed experimenting with a job guarantee is a clear indication that some in the party think it could be a political winner as well as a policy advance. In the wake of the 2016 election, liberal commentators have latched onto the job guarantee as a way to forge a cross-racial working-class coalition. They need a plan that appeals to both to the white Wisconsin and Michigan voters who switched from Obama to Trump and to black and Latino workers left behind by deindustrialization. The ideal plan would both improve conditions for lower-income Americans while supporting Americans’ strong intuition that people should work to earn their crust. “A federal job guarantee is both universal—it benefits all Americans—and specifically ameliorative to entrenched racial inequality,” Slate’s Jamelle Bouie noted. “The job guarantee asserts that, if individuals bear a moral duty to work, then society and employers bear a reciprocal moral duty to provide good, dignified work for all,” Jeff Spross added in the influential center-left journal Democracy. “If Democrats want to win elections, they should imbue Trump’s empty rhetoric with a real promise: a good job for every American who wants one,” wrote Bryce Covert in the New Republic. “It’s time to make a federal jobs guarantee the central tenet of the party’s platform.” As with any sufficiently big idea, however, a job guarantee comes with risks. It would be expensive, for one thing. Paul, Darity, and Hamilton’s plan, as outlined for the CBPP, would cost $543 billion per year, they estimate. For context, that rivals the size of Medicare, which will cost $707 billion this year, and the size of the defense budget, which totals $622 billion. The Center for American Progress has proposed a more modest $158 billion per year government jobs plan, but it wouldn’t actually guarantee a job to all who sought one, as Paul, Darity, and Hamilton have proposed. Booker’s bill doesn’t have an estimated cost yet. Second, it’s not clear what exactly participants in a job guarantee scheme would do. Most proposals are somewhat vague on this point, mentioning everything from child care to infrastructure. The Booker bill leaves most of this up to localities to figure out. Cities, counties, and other areas would get a chance to apply for the program, and the Department of Labor would be required to pick a mix of urban and rural areas to try the idea out in. Once the program was up and running, the vast majority of jobs offered would be in the public sector, and participants who need more skills development would be offered up to eight weeks of training. Positions meeting “a list of national job priorities … that shall include child care, care for seniors and individuals with disabilities, [and] infrastructure activities” would be encouraged, and private sector employers would benefit from the Work Opportunity Tax Credit, a benefit which currently gives employers a few thousand dollars toward wages when they hire veterans, welfare or food stamp recipients, ex-felons, and other people with barriers to employment. But beyond that, the bill doesn’t give many specifics. A job guarantee would ideally provide for jobs that are nice to have but could be nixed when the economy improves and the program shrinks in size. Those are pretty hard to identify. If a job guarantee were enacted in a recession, and many of the enrollees became child care providers, what happens when the economy improves and workers find jobs in the private sector? It wouldn’t be tenable to eliminate a universal child care program because the economy improved. Nor, if the program employed bus drivers, would it make much sense to cut bus routes. The Booker bill requires that the program be subject to an ongoing rigorous evaluation, which would help determine how well-equipped local governments are to make those kinds of distinctions. That would be a welcome addition to the research literature, as there’s some past evidence suggesting programs like this struggle to succeed. Berkeley economist David Card recently conducted a meta-analysis of more than 200 evaluations of programs meant to boost labor markets, along with fellow economists Jochen Kluve and Andrea Weber. While they found a variety of impacts of different programs, one constant was that public employment programs that simply hired people directly performed worst. “Public sector employment subsidies tend to have negligible or even negative impacts at all horizons,” the study concludes. “This pattern suggests that private employers place little value on the experiences gained in a public sector program.” One reason, they suggested, was that the programs did nothing to help build skills that would make participants more employable. A job guarantee, though, would be on an entirely different scale, and the Booker bill would enable an evaluation that could determine if its effects are more like the successful Indian program or the failed efforts that Card, Kluve, and Weber studied. With Donald Trump in the White House and Republicans in charge of both houses of Congress, it’s doubtful Booker’s bill will become law anytime soon. But that’s not really the point. Until his proposal, the closest thing there was to a job guarantee bill in Congress was Rep. John Conyers’s (D-MI) Humphrey-Hawkins 21st Century Full Employment and Training Act, named after a largely failed 1970s bill pushed by civil rights activists and unions to enact a job guarantee. But the bill fell short of establishing a formal guarantee, and Conyers has since resigned from Congress in a sexual harassment scandal. Booker has put the idea back on the Democratic agenda, and in a form that could be readily passed at relatively minimal cost (given the pilot program nature) when Democrats are next in power. That’s an important step, and one that significantly improves the odds that the idea will be part of the next Democratic president’s to-do list.

Building on these models, we propose the creation of an open, online networking platform for rigorous policy pilots, which could be hosted at, for example, PolicyPilots.gov or PolicyPilots.org. At the site, government agencies could publish requests for proposals on rigorous policy pilots (prospective or retrospective) in particular subject matter areas. Academics could post their own research skills and interests, and respond to agency requests that interest them the most. Likewise, agencies could reach out to relevant researchers who might be a good fit for projects they are considering.

Listings on the platform could be required to provide much of the information that potential partners need to determine fit and speed the match process. Agencies could indicate the parameters of the evaluation opportunity—for example, the question at its heart and why it matters, the authority to carry it out, the hypothesis and theory of change, testing methods that support causal inference, relevant evidence, and resources—consistent with the “MATTER” framework one of us has developed for proposing a rigorous pilot. Researchers could also post their areas of interest, current projects, willingness to travel, and resource needs. The high quality and ready availability of state and local data in many jurisdictions, as well as the skills of academic researchers, could make collaborations especially desirable.

(Colleen V. Chien, Justin D'Atri Visiting Professor of Business Law, Columbia Law School; Professor, Santa Clara University School of Law; 2013-2015 White House Senior Advisor, Innovation and Intellectual Property, “Rigorous Policy Pilots: Experimentation in the Administration of the Law, July, 2019, “104 Iowa L. Rev. 2313,” pg lexis//um-ef)

If the premise of this Essay - that using rigorous piloting is legal, feasible, and desirable - is correct, then it is incumbent upon policymakers to think about how to make it easier. The descriptions above and discussions at the companion workshop to this Essay reveal a number of suggestions for doing so: Clear obstacles to federal agency RCTs imposed, for example, by the Paperwork Reduction Act ("PRA"), reduce burdens associated with receiving feedback from stakeholders in general imposed through the Paperwork Reduction Act, create a networking mechanism for connecting federal agencies and academics interested in collaborating on rigorous policy pilots, provide dedicated capacity within government to support policy development through experiments (similar to what OES does for program implementation), make it easier for agencies to get access to the resources they need to carry out pilots, create safe harbors for agencies around the disclosure of procedurally adequate experimentation and its results, support evaluation and piloting mandates within statutes.

In recent years, citizens and lawmakers have become increasingly enthusiastic about the adoption of evidence-based policies and programs. Social scientists—aided in part by an increased use of field experiments—have delivered evidence of countless interventions that positively impact peoples’ lives. And yet these programs, when expanded, have not always delivered the dramatic societal impacts promised.

Should we assume, then, that we have been too quick to embrace evidence-based policymaking?

(John A. List is the Kenneth C. Griffin Distinguished Service Professor of Economics and co-founder of the TWC Center for Early Learning + Public Health at the University of Chicago., “How Can Experiments Play a Greater Role in Public Policy?,” pg online @ https://www.theregreview.org/2019/11/18/list-how-can-experiments-play-greater-role-public-policy/ //um-ef)

In recent years, citizens and lawmakers have become increasingly enthusiastic about the adoption of evidence-based policies and programs. Social scientists—aided in part by an increased use of field experiments—have delivered evidence of countless interventions that positively impact peoples’ lives. And yet these programs, when expanded, have not always delivered the dramatic societal impacts promised. Should we assume, then, that we have been too quick to embrace evidence-based policymaking? Not at all. Instead, we should acknowledge what is missing: a proper understanding of how, why, and when promising results can be delivered at scale. Unsurprisingly, when an intervention moves from the research setting to population-wide implementation, the magnitude of the treatment effect—or the benefit-cost estimate—might change. The change can be negative or positive and is influenced by a host of factors. In a series of new papers, my colleagues and I refer to this phenomenon as the “scale-up effect.” To understand what causes the scale-up effect, we approach the problem through the lens of economics. By recognizing the incentives of various actors in the scientific market of knowledge creation, we identify three key ingredients to understanding the scale-up effect: What constitutes actionable evidence? Policymakers should carefully evaluate the results of policy experiments, and should avoid scaling programs before there is sufficient evidence of efficacy. The scale-up effect becomes evident when an intervention tested in a small-scale randomized controlled trial suffers from an inference problem, such as a false positive, and then is implemented at a larger scale. How do the properties of the population affect scaling? Threats to scalability emerge when an experiment’s participants are not representative of the policy population. A non-representative participant pool can be caused by several factors, including researcher incentives. In a competitive scientific marketplace, researchers may be more likely to choose a subject pool with a greater chance of showing a treatment effect than would a random sample or the policy population. For example, after research found that fortified salt was successful at reducing anemia rates, a program based on that research rolled out to broader Indian populations had no effect. Why? Because it turns out the initial studies sought out adolescent women, not a representative population. Although the positive effect remained intact for adolescent women, it was absent for other groups. How do the properties of the situation affect scaling? Threats also emerge when the experimental situation is not representative of the policy domain. Properties of the situation include specifics of the program, correct dosage, correct delivery, and implementation costs, as well as spillover effects—unanticipated consequences on participants receiving the treatment, individuals in the control group, or even people who are not participating in the experiment at all. For example, the Student Teacher Achievement Ratio (STAR) project reduced class sizes in early grades in Tennessee, delivered promising results. But when California tried to replicate its success by reducing class sizes statewide, it failed to do so. Why? Because implementing the program at this scale required California to hire many additional teachers, and this led to the hiring of teachers with little teaching experience compared to those in the initial STAR project—a significant difference in the situation. Careful consideration of these three categories of major threats to scalability leads to a set of recommendations for policymakers to reduce the scale-up effect and better understand which policies will scale effectively. PROPOSAL 1: Before advancing policies, the post-study probability should be at least 0.95—meaning that there should be at least a 95 percent likelihood that the experimental research finding is true. In our experience, some decision-makers in government and the private sector wish to rush new insights into practice before they are fully tested. This can contribute to the loss of impact at scale. Fortunately, an initially low post-study probability from a single study can be raised substantially if the initial positive findings pass as few as two or three independent well-powered replications, which usually produces a post-study probability of 0.95 or higher. PROPOSAL 2: Reward scholars for prioritizing replication in their research. Powerful incentives such as tenure decisions and public grant money should inspire scholars to attempt to replicate others’ findings, to produce initial results that do independently replicate, and to report on null results—especially “tight zeros”—as such results contain valuable policy information. A “tight zero” is a research study that shows with great accuracy that the intervention does not improve outcomes. Proposal 2 is a call to policymakers, funders, and the academic community to begin to recognize the import of replication studies. PROPOSAL 3: Leverage multi-site trials to learn about the variation of program impacts across both populational and situational dimensions. By using appropriate variation in individual-specific characteristics, multi-site trials can provide empirical information about why effects might not scale and give empirical hints about where more research is needed before scaling. PROPOSAL 4: Use of appropriate technology should be encouraged to promote standardization across experiments. PROPOSAL 5: When scaling a program, the original scientist should participate on the implementation team to enhance fidelity, to teach policymakers why the result occurs, and for general consultation. PROPOSAL 6: Policymakers must understand the non-negotiable components of a program that are required for the program to work, as opposed to negotiable components that can be altered without affecting the results. They should treat non-negotiable components as necessary conditions from scientists before scaling a program based on their research. PROPOSAL 7: When the program is actually scaled, follow-up studies should use the correct empirical approach to measure efficacy, and continuous measurement should be a priority. The best approach to estimating the effects of the program at scale is to perform a large-scale randomized controlled trial. If this approach is untenable, then policymakers should adopt an empirical approach that allows stakeholders to measure efficacy without unrealistic assumptions. These seven proposals are primarily intended for policymakers and program implementers. But researchers also have an important role to play in addressing threats to scalability, and in some cases they can take preemptive steps to avoid inadvertently suffering from them. Our theoretical framework begins to tackle what we consider the most important question facing evidence-based policymaking today: How can we combine economics with the experimental method to inform policy at scale? We hope this framework helps to explain the potential causes of both successes and failures in scaling. Our goal is to empower policymakers and practitioners to make more informed decisions. We also aim to educate and encourage funders of research, both public and private, to support and encourage studies that embrace this approach to the science of scaling. Providing insights into how results scale to the broader population is critical to ensuring a robust relationship between scientific research and policymaking and a bright future for evidence-based policy. The entire science-based community—from scholars to funders to policymakers—must join forces to tackle the weakest link in successful evidence-based policy: the scale-up effect.

(Colleen V. Chien, Justin D'Atri Visiting Professor of Business Law, Columbia Law School; Professor, Santa Clara University School of Law; 2013-2015 White House Senior Advisor, Innovation and Intellectual Property, “How Federal Agencies Have Used Rigorous Policy Pilots to Learn,” pg online @ https://www.theregreview.org/2019/12/06/chien-federal-agencies-used-rigorous-policy-pilots-learn/ //um-ef)

As candidates for the 2020 election continue to pitch their big ideas, perhaps the single biggest question in the minds of voters is, “Will it work?” This essay series has explored how rigorous tests and evaluation, widely used in the private sector, can be used to discover what works in law and policy. Why now? Calls for rigorous evaluation of public policy are hardly new and neither are the challenges. Implementation or contextual factors can make data-driven results difficult to replicate. History is replete with examples of politics or ideology trumping evidence. But even in relatively stable, uncontroversial areas of law, governmental actions are not always taken with an eye toward policy learning, because rigorous policy piloting is perceived to be unfair and possibly illegal, hard, and often not worthwhile. In a recent paper, I surveyed numerous examples of policy piloting across federal agencies and in case law and reached the opposite conclusion: that rigorous policy piloting is in many cases feasible and worthwhile, and that law does not stand in the way. Before describing some of these examples, it is worth taking stock of some of the reasons motivating the greater uptake of experimental approaches in agencies. First, the costs of rigorous evaluation are going down. Helpful factors include the advance of open data and administrative data sharing, reductions in the cost of outcome tracking, and the growth of policy lab collaborations. Second, the broader embrace of agile approaches, used to develop software iteratively, have contributed to a cultural shift in government and a greater, bipartisan willingness to test and learn. President Barack Obama’s 2013 management agenda called for “experimentation and innovation” to produce a “smarter, more innovative, and more accountable government.” Last year, President Donald Trump’s agenda declared that “smarter use of data and evidence is needed to orient decisions and accountability around service and results.” The landmark Evidence Act, described earlier in this series, will make it easier for agencies to do so. Against this backdrop, a number of agencies are already using pilots extensively to address open questions in law and policy. For example, the U.S. Patent and Trademark Office’s pilot program of randomized trademark audits has helped document inaccuracies in the trademark register, support a consensus among stakeholders, and inform strategies to strengthen the integrity of the register. The Office has also used pilots to drive continuous improvement in trademark and patent quality. The U.S. Department of Labor’s internal collaboration across 12 operational sub-agencies has helped the Labor Department identify agency learning goals and research questions, fund projects to investigate them, and evaluate the results by prioritizing independence, rigor, transparency, relevance, and ethics. The Office of Evaluation Sciences (OES) within the U.S. General Services Administration has implemented dozens of rigorous, controlled pilots leveraging behavioral science insights to improve program implementation in areas ranging from retirement security to economic opportunity. Controlled tests like those carried out by the OES can provide strong insights into the impacts of varying agency behavior. But rigorous policy pilots are not limited to randomized control trials. For example, the Office of the Investor Advocate at the U.S. Securities and Exchange Commission has used a variety of approaches—nationally representative surveys, behavioral choice experiments, and large qualitative studies—to advance its mission of providing evidence to support investors and Commission deliberations. The U.S. Food and Drug Administration’s (FDA) precertification pilot program for software-based medical devices shows how much can be learned from even a handful of case studies. As digital health products become increasingly ubiquitous, FDA’s pilot aims to test a streamlined regulatory framework for providing timely evaluation, pre-certification, and monitoring of software health products, all while ensuring safety. Although just nine companies are in the pilot, the data and feedback that they provide will be used to improve and refine the regulatory framework, and, later, to inform legislation. As the FDA pilot shows, when regulating emerging technologies about which there is much speculation, pilots are a good way to generate hard data. For example, the Federal Aviation Administration’s drone pilot program invited “visionary participants” from state, local, and tribal governments to conduct a variety of advanced operational tests under controlled criteria, so that the agency could accelerate its development of policies to advance both innovation and safety. In another emerging technology area, the Office of the Assistant Secretary for Research and Technology within the U.S. Department of Transportation is funding connected vehicle pilots in three cities to gather data needed for appropriate policy. On the local level, cities across the country are implementing pilot programs to gather data for local policymaking needs, and states are passing laws that encourage pilots. As the U.S. Court of Appeals for the D.C. Circuit has said, in some situations “a month of experience will be worth a year of hearings.” These examples show that rigorous policy piloting is possible and worthwhile. Earlier pieces in this series have described ways of making it easier, more routine, and successful. To encourage successful piloting from the outset, proposals include establishing a PolicyPilots.gov hub to connect academics and other potential evaluators with agencies running pilots, pre-registering experiments to reduce false positive results, setting aside resources for evaluation, and applying a learning mindset to rulemaking. During pilots, streamlining institutional review board processes and Paperwork Reduction Act requirements would help experiments operate more efficiently. And finally, randomizations should be carried out with transparency and integrity to enhance government credibility. All of these proposals share a common thread: when agencies know what works in policy, everyone benefits.

(Todd, Attorney Advisor at the Administrative Conference of the United States, “The Evidence Act and Regulations,” pg online @ https://www.theregreview.org/2019/12/03/rubin-evidence-act-regulations/ //um-ef)

Two recent recommendations from the Administrative Conference of the United States (ACUS) call for agencies to promote a “culture of retrospective review” by planning for evaluation of a rule as it is being written. Such an evaluation includes defining how success will be measured, how the necessary data will be collected, and what kind of experimental methodology will be employed to draw valid causal inferences. Drawing on these two ACUS recommendations, Cary Coglianese and I have previously argued that having agencies create evaluation plans when they develop new regulations could help strengthen a culture of retrospective review of regulation. Just as society demands that pharmaceutical companies test how well their medical interventions work, regulators also should be expected to plan for and test how well their policy interventions work. In crafting an evaluation plan, an agency should develop a hypothesis about how its new regulation will work, specify the data it will need to collect to prove or disprove the hypothesis, and specify the methodology it will use to test the hypothesis (such as a randomized control trial or quasi-experimental method). In just the past year, Congress has taken action in exactly the direction that ACUS had recommended and Coglianese and I had urged. Both the House and Senate overwhelmingly passed the Evidence-Based Policymaking Act of 2018, and it was signed into law. This Evidence Act and an accompanying Office of Management and Budget (OMB) guidance call on agencies to apply to their programs the same basic framework laid out in the ACUS recommendations and our recent article. Under the Evidence Act, all agencies subject to the Chief Financial Officers Act must create “learning agendas” and “evaluation plans.” Agencies not subject to this Act are also strongly encouraged to do so. A learning agenda is simply a list of research questions for which the agency intends to conduct an evaluation to answer. For example, agency managers might ask: How successful are our regulations in promoting national-level environmental improvement? Or they might ask: How can we change our internal human resources process to shorten time periods for applicants? Evaluation plans are where the rubber meets the road. Agencies are required to identify from their learning agendas all “significant evaluations” and develop and publish a plan for those evaluations. Whether an evaluation is significant enough to warrant a published plan is up to the agency, but the agency must consider factors such as the importance of the program to the agency mission, the size of the program in terms of the number of people affected, and the extent to which an evaluation will fill an important knowledge gap in the program. The Act and accompanying OMB guidance do not specifically state that an agency must create an evaluation plan for a regulation. Agencies are left to decide what constitutes a “program” to be evaluated, as long as agencies consider importance, size, and knowledge gaps. Many regulations—for example, economically significant environmental and health regulations—affect all or nearly all people in the country. For this reason, it is difficult to imagine a scenario in which evaluation plans are not created for at least some regulations. Required evaluation plans must include at least the following five elements: Questions to be answered: In discussing the key questions, evaluation plans need to describe a program’s purpose and objectives and discuss how the program is linked to its intended effect. Each agency is encouraged to discuss any evaluation activities that relate to its entries in the Unified Agenda, recognizing that these activities need to occur well before the development of economically significant regulations. Information needed for evaluations: Agencies must note whether they will undertake new information collection requests or if they will use existing information. Methods to be used: Agencies must describe the evaluation design. For example, will they use a randomized control trial or a quasi-experimental method? Agencies could benefit from reviewing the ACUS recommendation Learning from Regulatory Experience as they explore some of the analytical and ethical advantages and disadvantages associated with each approach. Anticipated challenges: Agencies should discuss any anticipated challenges to completing their evaluation plans. For instance, agencies may find that, in attempting to collect data, they will need to be mindful of the Paperwork Reduction Act. Dissemination: Agencies must propose how they will use the evaluation results to inform policy making. Agencies do not need to create an evaluation plan before a program is put in place. An agency can create evaluation plans solely for existing programs. Agencies, however, would do well to heed the advice of ACUS’s recommendation on retrospective review, which encourages them, prior to issuing new regulations, to establish a framework for assessing these regulations in the future and then to include portions of the framework in the preamble. Creating such an evaluation plan before a rule is created saves the agency time and energy later. This is because prospective evaluation plans can ensure that the agency has identified the correct data it needs to evaluate the rule, identified potential roadblocks to obtaining that data, and designed the rule to allow for valid causal inferences. Once a rule is in place, it is still possible, although far more difficult, to design a valid plan for determining its effectiveness. Of course, it is not possible for an evaluation plan to be created for every program or regulation. The Evidence Act and OMB guidance recognize that agencies have limited resources. With respect to rules, for example, agencies still must satisfy requirements that have long predated the Evidence Act, including those under Executive Order 12,866 and various statutes. Evaluation can be very time intensive. Regulatory impact analyses often consist of hundreds of pages of economic, engineering, and other technical analyses. Fortunately, the ACUS recommendation on retrospective review lays out 11 factors that agencies can consider in deciding how to prioritize those rules for which to create an evaluation plan. ACUS recommends that agencies take into account factors such as: the agency’s degree of confidence in its initial estimates of regulatory benefits and costs; the likelihood of increasing net benefits and the magnitude of those potential benefits; and the likelihood of attaining the statutory objective. Whether an agency chooses to use these criteria or develop its own separate or additional criteria, it will need some way of prioritizing the rules for which it will create evaluation plans. By requiring agencies to create evaluation plans for their programs, the Evidence Act empowers agencies to think rigorously about whether the solutions they put forward adequately address the problems they are designed to remedy. By publishing these evaluation plans and soliciting input from the public, agencies can refine their research methodologies and develop better-informed programs, including their regulatory programs. Thanks to the requirements of the Evidence Act, the future of evidence-based regulation has never looked brighter.

Project management is a long game. It can take time to see the results. But one way to test things out on a small scale before knowing if it will be a success is to run a pilot project. A pilot project enables you to manage the risk of a new idea and identify any issues before you fully commit resources.

Take launching a product in a new market, for example. A pilot project would identify and address any key issues – including resources, product-market fit, and customer feedback – before rolling it out to a larger audience.

(Colleen V. Chien, Justin D'Atri Visiting Professor of Business Law, Columbia Law School; Professor, Santa Clara University School of Law; 2013-2015 White House Senior Advisor, Innovation and Intellectual Property, “Rigorous Policy Pilots: Experimentation in the Administration of the Law, July, 2019, “104 Iowa L. Rev. 2313,” pg lexis//um-ef)

Each of these steps is non-trivial, requiring a deep understanding of the powers and authorities of the relevant agency, the context in which it operates, the content and footprint of high priority questions, and the data sources from which meaningful metrics can be developed. Though the implications or recommendations section of a law review article or report often get the least amount of space, designing and proposing a pilot often requires novel contributions that support not a fixed policy "product," but a dynamic policy "process" for testing and refining a policy idea before it is fully [*2350] implemented. A broader recognition of the importance and originality of this "translational work" - by law review and other editors - would shift the focus of academic writing from "admiring the problem" 215to "solving the problem."

5 Social security as poverty alleviation and social protection programmes

Ideally, social security is a permanent social welfare structure which aims at a piecemeal and steady incremental improvement in the quality and standard of living of the citizenry. There is no gainsaying the fact that, in most developing communities such as Nigeria, there is a causal relationship between the absence of social security on the one hand, and poverty, on the other. However, the government initiatives which focus on temporary or permanent poverty eradication, are not social security measures, but are transient steps taken to form a suitable foundation on which social security programmes may be based. For example, in 2004 a survey of policies and interventions of various administrations on issues of poverty revealed 28 federal projects and programmes with poverty-reduction aims.73 These include community banks; the Family Support Programme; the National Directorate of Employment; the Peoples' Bank; Better Life for Rural Women; and the Directorate of Food, Road and Rural Infrastructure. These are some of the programmes that were established to address various manifestations of poverty, such as unemployment, lack of access to credit and rural and gender dimensions of poverty.

And, “substantial” means the increase must be definite. Potential future increases are not topical. Words and Phrases, 1964, accessed online: “Substantial means not concealed, denoting that which not merely can be, but is supposed to be, potential, certain, absolute, real at the present time.”

The words “outward, open, actual, visible, substantial, and exclusive,” in connection with a change of possession, mean substantially the same thing. They mean not concealed; not hidden; exposed to view; free from concealment, dissimulation, reserve, or disguise; in full existence; denoting that which not merely can be, but is opposed to potential, apparent, constructive, and imaginary; veritable; genuine; certain; absolute; real at present time, as a matter of fact, not merely nominal; opposed to form; actually existing; true; not including admitting, or pertaining to any others; undivided; sole; opposed to inclusive