Pilot programs are prerequisite to effective public-policy trajectories.

Policy pilot evaluations often adopt a relatively simplistic approach of considering whether or not a pilot has “worked” against particular outcome criteria. We have shown that this represents a limited understanding of the purposes and effects of pilots. We have brought together literature on policy success with that on policy pilots to generate a framework within which empirical evaluation findings can be synthesized with analysis of the wider policy landscape to consider pilot success in a more nuanced and multifaceted way. While no such framework can be completely comprehensive, and judgments as to “success” in each category will be contingent, provisional, and potentially arguable, we would suggest that the most valuable aspect of the framework is its focus on explicit delineation of different dimensions of success and on the declaration of the perspective being adopted. While different commentators may disagree with particular judgments in each cell of our summary table, the criteria by which we are judging and the specific aspect of the pilot being judged are clear, providing a more nuanced evaluation framework and facilitating constructive discussion.

Furthermore, we believe that explicitly considering pilots across all of these dimensions holds promise in supporting the design of more comprehensive and nuanced evaluation programs which move beyond a simplistic attempt to demonstrate “what works.” We have also shown how the explicit separation between program and political elements of success allows interrogation of the antecedents of particular policy decisions, facilitating deeper understanding of contemporary policy trajectories. This in turn supports a more nuanced understanding of later policy developments. For example, we suggest that early political claims to the success of the Vanguard program may have inhibited longer-term consideration of the practical steps required to facilitate program success, with potentially significant implications for NHS managers seeking to implement new approaches to service integration. Moreover, we would suggest that our framework facilitates a more nuanced understanding of pilots, looking beyond stated rationales, and encouraging evaluators and others to explicitly consider the extent to which particular pilots do indeed represent a rational attempt to test policy ideas, or whether they in fact represent an approach to implementation. “Success” can then be judged against these more complex objectives. We invite others interested in this area of research to consider the applicability of this approach in other fields, with a particular focus upon issues of temporality and methods to identify criteria by which political success might be judged.