Empirics disprove – AM&E itself is waste
Görgens 5 Marelize Görgens, World Bank, HIV/AIDS Monitoring and Evaluation consultant, “Monitoring and Evaluation Systems for HIV/AIDS: chasing data or challenging paradigms?” November 2005, https://www.researchgate.net/publication/291347597_Monitoring_and_Evaluation_Systems_for_HIVAIDS_chasing_data_or_challenging_paradigms /GoGreen!
Within the international development arena, the past few years has witnessed a move towards measured, progress-focused development, as illustrated by the birth of terms such as “effective aid ” and “managing for development results ” at various international forums. For example, at a global meeting of development partners in 2004, , the discussions centred on how to strengthen an international partnership around the concept of ’managing for development results’. This upped the stakes for monitoring and evaluation (M&E), in the sense that the Joint Memorandum, which was issued by all these global partners at the end of their meeting, focused not only on M&E processes itself, but also on using results for improvement and decision-making.
“[We will] rely on – and strengthen – countries’ monitoring and evaluation systems to track progress and assess outcomes. As agencies with regional or global reach, we pledge to better distill the lessons of countries’ experiences and disseminate knowledge about what gets results in different country contexts.”
Joint Memorandum, issued by all global development partners at the end of the 2nd Roundtable on Managing for Development Results
Despite pressure from international development partners to measure and quantify results, M&E has remained the ugly duckling of development. Efforts to incorporate it into practice have met with a combination of apathy, resistance, and half-baked efforts that primarily consisted of throwing together a few indicators and defining how the data would be collected.
Developments in the field of HIV/AIDS monitoring and evaluation over the last 10 years has illustrated the challenges of monitoring and evaluating of development efforts in general, and raised some uncomfortable, close-to-home questions for the development M&E profession.
In Africa in particular, HIV/AIDS M&E is not only important because of international pressure on managing for development results, but also because of the scale of the virus’s impact. HIV/AIDS has ravaged Africa. A continent-wide Social Impact Assessment would have revealed that it has destroyed families, torn apart communities, paralysed the health sector, crippled the agricultural and education sectors, hampered government’s efforts to provide effective services to its people, and caused indescribable long-term trauma and grief. The extent and reach of its impact over such a short period of time is forcing national AIDS coordinating structures to focus, maybe for the first time, on ways to find real, effective, efficient, and replicable solutions.
At the international level and at the country level, the response to HIV/AIDS has moved in the past 10 years from a health sector-driven to a multisectoral response. In parallel, HIV/AIDS monitoring and evaluation has evolved from exclusively epidemiological surveillance in the 1980s to multi discipline approaches to HIV/AIDS M&E.
B What are the challenges?
HIV/AIDS M&E cannot be a disjointed collection of evaluation studies and monitoring processes, but needs one integrated HIV/AIDS M&E system, managed by a National AIDS Commission (NAC) MG comment: some countries do not have NACs, so it may be better to refer to it as the national AIDS coordinating structures. This is because the NAC has the mandate to coordinate and manage the national HIV response and can only do this if it has accurate and relevant data when decisions need to be made. It is also because it was recently acknowledged, internationally, that one M&E framework at country level is one of the core principles of effective management of the HIV response .
Such an HIV/AIDS M&E system needs to combine a vast variety of M&E processes: from biological sentinel surveillance (an epidemiological “tripwire” used to detect trends in HIV infection); to routine collection of management information about funding for HIV interventions in the public sector, private sector and civil society; and ethnographic-type studies about sexual risk behaviour and the abuse of right of vulnerable children, the elderly and persons living with AIDS.
HIV/AIDS M&E requires that, in order to assess the extent and nature of HIV service coverage, the same output-level, programme-monitoring data be collected from all implementers of HIV interventions on a routine basis. The health sector in Africa has struggled to establish an effective management information system in its own sector – NACs are now obliged to set up a such a routine system across all government ministries, the private sector, and the entire civil society. This is a vast undertaking, especially given that some of these organisations have never had to report to government before. Another challenge is that the HIV response is increasingly becoming decentralised. This adds a layer of reporting and could potentially cause delays in data collection.
HIV interventions range dramatically in content, focus, and implementation methodology. For example, interventions focusing on establishing partnerships for HIV will have vastly different monitoring and evaluation approaches to programmes that aim to keep educationally marginalised children in school.
The impact of HIV has necessitated that there is a real focus on using results for decision-making and programme improvement because countries in Africa heavily burdened by HIV/AIDS cannot afford not to.
B What can be done?
Addressing these challenges requires a systemic approach: we need to focus on designing and operationalising national HIV/AIDS M&E systems, rather than new evaluation studies.
Figure 1 illustrates such a systemic approach to HIV/AIDS M&E. It shows the relationship between indicators (A), data sources (B), information products (C) and the project stakeholders (D). These four components will only work in unison if it is backed up by strong management.
Figure 1: Illustration of Systemic Approach to HIV/AIDS M&E
At the risk of revamping an old development phrase, a systemic approach to HIV/AIDS M&E require those involved in the M&E profession to change. It requires a paradigm shift away from conducting piecemeal, complicated, and specialised monitoring or evaluation studies and towards thinking about monitoring and evaluation in an operational way, focusing on synergies, sometimes at the expense of individual academic accolades.
B The Way Forward
Such an approach requires that M&E professionals:
•Learn new skills: they need to become good budgeters, M&E work plan developers and M&E strategists.
•Move away from focusing on specific processes and towards integrated monitoring and evaluation systems, which re-examines the practicalities of collecting output data.
•Use existing tools in a new way: For example, Social Impact Assessments have, to date, primarily been used to assess the impact of planned developments (e.g. assessing the social impact of a new mine on a community’s livelihood). The tools in the quiver of the Social Impact Assessment process (including trend extensions, computer modelling, consulting experts, focus group discussion and questionnaires) could be adapted to even suit assessments of the unplanned, catalytic, and widespread impact of HIV and AIDS on different sectors of society.
•Assess the cost-benefit trade-off in different monitoring and evaluation tools and processes to find, recommend, and use those processes that are most useful for the most reasonable cost
•Be prepared to sacrifice some professional autonomy for the sake of synchronisation. In one country in Africa, in 2004, over 40 assessments relating to HIV/AIDS M&E were done, often replicating each other.
•Understand how HIV/AIDS M&E systems should be managed to yield the best possible success; it implies advocacy for M&E, capacity building in M&E and communications about M&E
•Find a renewed focus to make sure that results are used to improve projects.
•Go back to the basics of data collection, and apply it with new vigour in a multi-sectoral, decentralised, developing and emergency context.
•Think of practical ways in which to focus on, promote and institutionalise the process of using results for decision-making. This means moving away from the typical thinking in which M&E systems have focused on indicators and data sources, with less focus on information products and dissemination to stakeholders (see Figure 2). It also implies avoiding a situation where stakeholder dissemination is done as an afterthought, primarily through a national level workshop.
Figure 2: Disproportionate focus on Indicators and Data Sources
The challenges of HIV/AIDS monitoring and evaluation systems cut to the heart of the monitoring and evaluation profession. Although we have always known that monitoring and evaluation needs to earn its keep (Patton, 2000), it is nowhere more evident than in the design and implementation of HIV/AIDS M&E systems. Every dollar that is spent on monitoring and evaluation is one that it not spent on HIV programme implementation. Within the context of HIV/AIDS, this can simply not be wasted.
HIVAIDS is an opportunity for our profession to prove M&E as a real, tangible and value adding part of development work, and as an opportunity for professionals to demystify monitoring and evaluation. We have a responsibility to do so, and there has never been a better time.