Search the archive.

But none of the experiments have (so far) ended with the implementation of basic income as a policy, by keeping the pilot going or rolling it out to wider populations. The International Public Policy Observatory (of which The Conversation is a partner), in collaboration with the Institute of Policy Research at the University of Bath, recently examined 38 experiments across Europe, North America and Asia (in countries that already had established welfare systems) and found three main reasons why the effort ended with the pilot.

1. Gaps in evidence

Determining the impact of a basic income requires comparing outcomes, both with the behaviour and attitudes of people beforehand (before the ‘treatment’ began) and with ‘control groups’, otherwise similar, who do not receive the ‘treatment’ or come under the policy. This is to ensure that outcomes are the result of the basic income and not a change coming from the outside.

While randomised control trials (RCTs) are currently in vogue, they have severe limitations when it comes to evaluating basic income. The principle behind RCTs, as the name implies, is that those receiving the ‘treatment’ should be selected ‘randomly’ from a wider population, and the control group should also be selected ‘randomly’. This would typically mean listing households in an area and drawing a sample, so that one family (or individual within a family) would receive the basic income while the family next door would not.

(Colleen V. Chien, Justin D'Atri Visiting Professor of Business Law, Columbia Law School; Professor, Santa Clara University School of Law; 2013-2015 White House Senior Advisor, Innovation and Intellectual Property, “Rigorous Policy Pilots: Experimentation in the Administration of the Law, July, 2019, “104 Iowa L. Rev. 2313,” pg lexis//um-ef)

At the same time, as described above, rigorous experimentation will neither remove the burdens of ensuring an adequate nexus to a legitimate governmental objective, nor absolve an agency of procedural compliance with notice and comment, nor shield agency from scrutiny. When the stakes are high and agency action is likely to have material consequences for stakeholders, experimental evidence, like other evidence, is likely to be contested and challenged. While on one hand, experimental evidence can create a shared experience, on the other hand, stakeholders and even agencies are likely to interpret evidence in the manner that advances their interests. 135Matters of institutional competence may come into play if the court is unsure how to evaluate an an agency's experimental design or interpretation of experimental results.

Budget Benefits – A pilot project can benefit the project budget in several ways. For example, if the pilot produces a deliverable, which can be “reused” when the concept or innovation is approved (and it is produced early), then there will be a budget benefit. In addition, budgetary numbers can be obtained by the project manager for the post-pilot era.

Scope Benefits – Any scope pre-gathering exercise is beneficial. For example, as a pilot is executed, participants have the opportunity to contribute requirements as the innovation or concept is deployed. Theoretically, this will assist with future deployment of the innovation or concept and control “runaway” requests during this time.

Collaborative “generic” non-rate case dockets that focus on broad topics and bring stakeholders together for non-litigated consideration of potential reforms have begun to include work on pilots.

Utilities and other frequent docket participants like consumer advocates are prepared for litigated proceedings, but “technology providers may come with data and not understand why it is questioned,” the Regulatory Assistance Project’s Sedano said. In generic dockets, like New York’s years-long Reforming the Energy Vision, or REV, regulators have helped participants “find shared interests” and “changed the way utilities work with stakeholders,” he recalled.

President Joe Biden says he wants to renew America’s democratic alliances. He has outlined a new foreign policy agenda, which aims to reinvigorate democratic values and provide stronger competition to rising authoritarian powers. His administration writes about fighting kleptocracy and climate change, conquering inequality, and standing up for human rights. There’s talk about a “Summit for Democracy” to promote the new agenda.

None of it can succeed, though, without an information environment that allows arguments to be won with evidence and allows citizens to deliberate about significant issues in a full, free, and fair conversation. Such an environment does not exist today in the United States or most other democracies.

Even if clients have a ton of work in their backlog, never jump into a partnership without testing the waters. A pilot project not only shows a client what it will be like to work with a new vendor, but it also reveals a client’s expectations and collaborative style. If one side proves to be a difficult partner, the two parties can scrap the arrangement with no hurt feelings and little time lost.

In most cases, though, pilot projects help clients and vendors set the stage for better partnerships. By limiting the scope and providing detailed estimates, vendors can avoid communication breakdowns and adapt to potential partners’ working styles. This process is especially important for more technical companies, where scope creep and minor details can quickly grow into major obstacles.

Rigorous policy evaluation often involves randomization, and both federal and state governments have used randomization for a variety of purposes for approximately two centuries. But in the particular context of random government audits, transparency of process is crucial—especially when non-compliance can have reputational effects.

One of the first recorded government randomizations was by states in the early 1800s. It involved the shameful removal of indigenous people from their land followed by distribution of that same land to white people via land lottery. Other randomizations over time have included diversity visa lotteries, the draft lottery for selective military service, and education lotteries to place students in public schools. Panels of judges are often randomly assigned to review cases, and administrators on some government boards are also randomly assigned to make decisions.

A limitation of current regulations is that they focus on technologies known to pose a risk to human life. Entirely new innovations or obscure technologies receive far less attention. If a particle accelerator might inadvertently set off a chain reaction that destroys all life, only their internal safety process and self-preservation instincts prevent them from taking risks on behalf of everyone. There is no international process for ensuring a new technology won’t end up being a black marble.

Avoiding black marbles is both a coordination problem and an uncertainty problem. In the long term, it’s not enough to get 90% of creators to refrain from exploring dangerous territory. Absent strong coordination mechanisms, future technological development suffers from the unilateralist’s curse. Even if no individual creator has ill-intent, those who believe their technologies prevent little risk will forge ahead, biasing the overall population of creators towards unsafe exploration.

Pilot projects are incredibly helpful when it comes to lowering the risk of implementing new tools and tech. Let’s take a closer look at what a pilot project is, plus how and why you should run one.

What is a pilot project?

Budget Benefits – A pilot project can benefit the project budget in several ways. For example, if the pilot produces a deliverable, which can be “reused” when the concept or innovation is approved (and it is produced early), then there will be a budget benefit. In addition, budgetary numbers can be obtained by the project manager for the post-pilot era.

Scope Benefits – Any scope pre-gathering exercise is beneficial. For example, as a pilot is executed, participants have the opportunity to contribute requirements as the innovation or concept is deployed. Theoretically, this will assist with future deployment of the innovation or concept and control “runaway” requests during this time.

(Colleen V. Chien, Justin D'Atri Visiting Professor of Business Law, Columbia Law School; Professor, Santa Clara University School of Law; 2013-2015 White House Senior Advisor, Innovation and Intellectual Property, “Rigorous Policy Pilots: Experimentation in the Administration of the Law, July, 2019, “104 Iowa L. Rev. 2313,” pg lexis//um-ef)

For example, in the Aguayo case described earlier, the Secretary's approval of a pilot program was made pursuant to 42 U.S.C. § 1315, which authorizes statutory waivers to support welfare demonstration projects. In reaching its decision, the Second Circuit considered the statute's objectives, the Secretary's consideration of the contested programs in view of the objectives and evidence, and the adequacy of the administrative record, including the agency's consideration and response to objections raised by the appellant. 115But the court also relied upon "the fact that the programs were of limited duration and would remain under ... on-going supervision" 116to endorse the setting of "a lower threshold for persuasion" for the approval of experimental, temporary programs. 117Finding there to be "no "clear error of judgment,'" 118the court specifically noted that "experience will be the best test of the reality of appellants' fears." 119

(Colleen V. Chien, Justin D'Atri Visiting Professor of Business Law, Columbia Law School; Professor, Santa Clara University School of Law; 2013-2015 White House Senior Advisor, Innovation and Intellectual Property, “Rigorous Policy Pilots: Experimentation in the Administration of the Law, July, 2019, “104 Iowa L. Rev. 2313,” pg lexis//um-ef)

Part II reviews relevant case law and finds that rigorous policy experiments raise no particular legal barriers to agency action. Many pilots will be uncontroversial because they do not require the mandatory, selective application of law or policy to the public. But pilots that do, including through randomization, have by and large been found by courts to pass constitutional muster on the basis that they further learning and other legitimate government objectives. The requirements that the Administrative Procedure Act ("APA") imposes will also vary depending on the design and features of a pilot but should not present special procedural hurdles to rigorous pilots nor elevated substantive hurdles to the use of the evidence they generate.

(Colleen V. Chien, Justin D'Atri Visiting Professor of Business Law, Columbia Law School; Professor, Santa Clara University School of Law; 2013-2015 White House Senior Advisor, Innovation and Intellectual Property, “Rigorous Policy Pilots: Experimentation in the Administration of the Law, July, 2019, “104 Iowa L. Rev. 2313,” pg lexis//um-ef)

Collectively, these cases confirm that the Constitution and APA present limited if any special barriers to rigorous piloting. No decision that I could find has held that experimentation rendered an agency rule per se arbitrary and capricious because it demonstrated a lack of sufficient information to select a course of action. To the contrary, courts are generally accepting of experimentation as a rational way of learning about a policy and value [*2335] randomization as an adequate, and potentially superior, 134way of distributing a benefit or a burden to a population.

(Colleen V. Chien is a professor at the Santa Clara University School of Law. Neel U. Sukhatme is a professor at Georgetown University Law Center and a visiting scholar at the U.S. Patent and Trademark Office., “A Proposal for PolicyPilots.gov,” pg online @ https://www.theregreview.org/2019/11/19/chien-sukhatme-proposal-policypilots-gov/ //um-ef)

We all want policies that work. But too often academics seeking opportunities to study important problems and policymakers seeking analytical muscle to evaluate policies cannot find one another. We propose a way to connect them. Researchers’ quests for knowledge and academic publications have recently sparked a “causal revolution” focused on uncovering the determinants of observed outcomes. The open data revolution has similarly kindled a strong demand for knowledge about how policies are working. Such knowledge can be generated by pairing academic researchers, with the requisite skills, interests, and training, with government agencies looking to enact evidence-based policies. These partnerships often yield important insights: For example, some of the most significant recent discoveries about intergenerational mobility have come from academic-government collaborations. But while the potential for collaboration between government and academia is great, so too are the obstacles. Academics often lack knowledge of political timing, program specifics, and the constraints of working within a government bureaucracy. They are also unaware of which government agencies are interested in rigorous policy testing and how that might overlap with their research. Moreover, concerns about the ability to publish research that is based on internal government data and a lack of clarity about data access and publishing can deter academics from pursuing such projects at the outset. Likewise, government officials might not know the capabilities of academic researchers who study issues that could have direct policy application. More fundamentally, agency officials often face difficulties finding interested and able empirical researchers, who are typically geographically dispersed. Indeed, looking for collaborators requires a heavy investment of time, with an uncertain payoff. Programs like the Presidential Innovation Fellowship—which permits talented individuals to pursue short “tours of duty” solving problems in government—are inherently limited in number and are not currently set up to support part-time collaboration. The good news is that a few existing models have achieved some success in connecting policy problems and resources to policy solvers, to the benefit of both. Challenge.gov is an online platform for all prize competitions and challenges issued by the federal government. Data.gov hosts open datasets released by the federal government. And ClinicalTrials.gov posts information on ongoing and completed clinical trials. Although these platforms are not specifically geared toward academics, a variety of government-sponsored research opportunities—for example, those provided by the Institute for Educational Sciences—allow researchers to apply for and complete multi-year projects, keeping in mind evaluation at the outset. Other networks like the Latif Jameel Poverty Action Lab, which focuses on randomized impact evaluations for poverty-related issues, as well as state policy labs like those in California and Rhode Island, connect and support public officials and researchers. Another model is to bring academics into agencies on a temporary basis. One successful version of this is the Edison Visiting Scholar Program within the Office of the Chief Economist at the U.S. Patent and Trademark Office, a visiting scholar position that one of us currently holds. The Office of Evaluation Sciences within the U.S. General Services Administration also embeds researchers in government agencies to work on projects that leverage behavioral insights. Building on these models, we propose the creation of an open, online networking platform for rigorous policy pilots, which could be hosted at, for example, PolicyPilots.gov or PolicyPilots.org. At the site, government agencies could publish requests for proposals on rigorous policy pilots (prospective or retrospective) in particular subject matter areas. Academics could post their own research skills and interests, and respond to agency requests that interest them the most. Likewise, agencies could reach out to relevant researchers who might be a good fit for projects they are considering. Listings on the platform could be required to provide much of the information that potential partners need to determine fit and speed the match process. Agencies could indicate the parameters of the evaluation opportunity—for example, the question at its heart and why it matters, the authority to carry it out, the hypothesis and theory of change, testing methods that support causal inference, relevant evidence, and resources—consistent with the “MATTER” framework one of us has developed for proposing a rigorous pilot. Researchers could also post their areas of interest, current projects, willingness to travel, and resource needs. The high quality and ready availability of state and local data in many jurisdictions, as well as the skills of academic researchers, could make collaborations especially desirable. Even if a government agency is not seeking a partner, it could still list planned pilots online to facilitate peer and third-party evaluations of methodologies and protocols before interventions are implemented, putting projects on the right track from the start. Research suggests that agency piloting is widespread but that the conditions needed for rigorous evaluation are not always present at the beginning. A PolicyPilots.gov platform could also facilitate the sharing of enabling documents, policies, and resources for supporting rigorous evaluation in ways that are consistent with good science while also respecting privacy and addressing ethical, legal, and fairness concerns. An obvious challenge with this sort of endeavor is getting a critical mass of participants. We propose a few strategies that could reduce this start-up problem. First, the proposed networking platform would benefit from a credible, neutral third-party hosting organization, such as the U.S. General Services Administration or the Partnership for Public Service. This would enable both federal and state government agencies and academic researchers to feel confident engaging with the site, posting opportunities, and accepting projects from it. Ideally, the third-party intermediary would also have built-in connections with both government and academia, which could help it boost the program at the start by connecting potential collaborators in its network. Second, to populate the site, the sponsoring organization should encourage agency officials to come forward and post short descriptions of existing and planned pilots on which they invite comment or collaborations. Recruiting a few government agencies—state and federal—to post their existing projects, visiting scholar opportunities, and requests for proposals on the website would encourage academic researchers to interact with the site. A platform, by itself, is not a panacea. The incentives need to be aligned, and even the value of failed experiments should be recognized. But even within existing parameters, our proposed platform has the potential to deliver broad benefits. Researchers would receive help finding agency partners and overcoming common barriers to rigorous piloting. Public servants would get access to expertise for advancing their agencies’ missions and learning agendas. And all of this would be in service of policies that work, to improve the lives of the American people.

Offering no-strings-attached monthly payments to Denver’s homeless population should result in less money going toward alcohol, drugs and other vices if programs in other cities are any indication.

Stockton, Calif., and Chelsea, Mass., tried initiatives like the one a Denver entrepreneur is proposing, and officials on both coasts say the money they’ve offered to low-income residents helped with stable housing, access to food and full-time employment.

Last October, a fire tore through the apartment complex in Stockton, California, where Laura Kidd-Plummer had lived for five years. Nearly a decade earlier, Kidd-Plummer, who will turn seventy this year, had retired from her job in the wardrobe department at the Oakland Coliseum, where she had worked for twenty-one years. She eventually moved to Stockton in search of cheaper rent. After the fire, she and her dog, Poopee, a Pomeranian-Yorkie mix, were left homeless. Ever since, she told me, “I’m just trying to keep my head above water.” She stayed in a motel for a couple of months, which was covered by insurance, and then with acquaintances. In May, she moved into a loft in North Stockton. She was able to make the deposit on the apartment using funds she has received as a participant in the Stockton Economic Empowerment Demonstration, or seed, a basic-income pilot program that has provided unconditional cash transfers of five hundred dollars per month to individuals over the last year and a half. Before seed, Kidd-Plummer “had credit cards and had to use them to eat,” she said, because she was only eligible for sixteen dollars per month through food stamps. Now she’s able to cover food. “I’ve always worked, so most programs I’m not eligible for,” she told me. “I didn’t expect to be chosen. When I got the letter in the mail, I was floored.”

The program, spearheaded by Stockton’s mayor, Michael Tubbs, was scheduled to end this summer: this month’s payment was slated to be the last. In late May, Tubbs announced that seed would be extended through January, 2021, in response to the economic strain put on participants by the coronavirus pandemic. While the idea of extending the program had been under discussion even before the spread of covid-19, Tubbs told me that current conditions made doing so a “moral imperative,” as many participants have lost work, and those classified as essential workers face increased risk. “covid-19 has put the focus on the fact that a lot of Americans live in constant moments of economic disruption, because the fundamentals of the economy haven’t been working,” he told me.

For government agencies, pilot programs are a great way to experiment with new technology. Frequently, there are questions around implementation, access and overall functionality. Implementing a pilot program allows an agency to identify these challenges early, develop synergy with their provider and define the service model that will be necessary for larger scale adoption. Our two-year contract with a major government agency (the “Agency”) is a useful example of how agencies can benefit from a pilot program as well as the best way to ensure its effectiveness. CDS was privileged to be awarded a contract to develop and implement a Relativity project to assist the Agency in responding to public requests for information, including requests under the FOIA and Privacy Act, as well as to assist in streamlining its workflow to reduce the backlog of overdue FOIA requests. To accomplish our mission statement, CDS leveraged Relativity as a SaaS in our FedRAMP authorized Cloud with great success.

Benefits of a Pilot Program

New evaluation methods, simulation and modeling tools, and evaluation techniques from other industries can be used to speed the assessment process and improve its generalizability. - Starting quickly and improving a pilot project over time can ensure that any problems are addressed quickly, instead of at the conclusion of the pilot process. - Pilots should anticipate how the project might spread and evaluate the initiative under a variety of real-world circumstances in order to be generalizable. - Local leadership and culture, in addition to resources and data support, are vital to successfully implementing change

Initiatives that require large changes to people, processes and technology can be difficult to do well, but taking small steps can be a good way to get started for many reasons.

We often talk about doing pilot projects in order to gain quicker learnings and have greater opportunities for long-term success. Despite this, there isn’t nearly as much written about exactly how to approach those pilot projects.