7 January 2024
By Katja Hemmerich
In this first monthly spotlight, we highlight new research relevant to key issues discussed by UN governance mechanisms over the next month. This month, we highlight recent research on evaluation and assessments of, and by, international organizations which illustrates that evaluations are important policy negotiation tools in multilateral contexts.
With the UNDP/UNFPA/UNOPS Executive Board holding its first formal session at the end of January, followed by the Boards of UNICEF, UNWomen and WFP in February, it is a busy month of informal preparatory consultations and decision-making at the formal sessions. During this process, member states on these Boards will be reviewing a significant number of audit and evaluation reports as well as plan for future evaluations. The New York based Board members will be considering thematic policy evaluations undertaken by UNFPA, UNICEF and UNWomen, as well as reports of the Board of Auditors in the case of UNWomen. WFP’s Executive Board, when approving the next round of Country Strategic Plans also reviews the latest evaluations of programming in that country. All this is in addition to the assessments, evaluations and audits that many member states, especially donor countries, have also undertaken themselves. A November 2020 study by the Global Public Policy Institute (GPPI) showed that between 2016 and 2019 the volume of humanitarian donor assessments increased by between 20% and 40% each year, resulting in a more than 200% increase over the three year period.
The proliferation of audits, evaluations and other assessments across the UN system appears to indicate a growing trend in the use of data and evidence to try and find agreement on UN programming across different stakeholders. In the fractured multilateral context that the UN is operating in, this can only be a good thing. But from a practical perspective, there are challenges. Delegates to UN governance bodies are increasingly overwhelmed with the amount of information they need to consider. Thematic evaluations can often cost hundreds of thousands of dollars to complete, and take significant time and effort from UN, member state and even NGO implementing partner staff. Our spotlight this month explores how evaluations are being used as a negotiation tool and how to leverage them constructively while reducing negative and unintended consequences.
The role of evaluation in the UN
The formal purpose of audits and evaluations is to promote accountability as well as organizational learning and improvement. As has often been indicated in discussion on UN financing, increased donor funding can only be expected if the UN is increasingly transparent and accountable (see for instance the the Dag Hammarskjold and Multi-Partner Trust Fund Office’s latest publication “Financing the UN Development System: Choices in Uncertain Times 2023”). The GPPI review of assessments by humanitarian donors, however, does not find a link between more evaluation and more funding.
“We considered the hypothesis that larger financial contributions lead to a higher number of assessments. The presumed causality was that the more funding donors provide, the more they require assurance to account for their contributions. If that were the case, the increase in donor assessments over the recent years would be explainable by the growth of the humanitarian sector’s funding volume as a whole. However, comparing donors’ financial contributions to OCHA, UNICEF, UNHCR, WFP, and the ICRC with the number of formal assessments conducted shows no obvious correlation.” - GPPI, Independent Review of Individual Donor Assessments in Humanitarian Operations, Nov. 2020
So how can the proliferation of evaluations, audits and other assessments be explained? A new study of centralized UN evaluations, i.e. institutional and thematic evaluations, highlights how they are being used as a tool for negotiation among member states, within UN organizations and between member states and the UN. This study, by Professors Eckhard and Jankauskus, as well as the GPPI review demonstrate that evaluations are used as tool to garner stakeholder support for strategic approaches and direction or to reconcile competing policy positions across decision-makers. Participants in both studies highlighted how this works in different contexts. At the governing body level, for instance:
"Interviewees talked about 'political struggles' among member states and how evaluation may help to counter single countries trying 'to defend their programs... regardless of the results.'" - S. Eckhard and V. Jankauskas, Explaining the political use of evaluation in international organizations, 2020
Several participants also highlighted that evaluations were a useful tool in bridging competing policy positions between member states from the Global South and donor countries.
Competing policy positions also exist within UN organizations and within donor governments. Profs. Eckhard and Jankauskas’ study also demonstrated that when senior leadership in UN entities sets the direction for evaluations, they are often used as a tool to overcome internal fragmentation and also reconcile the leadership’s political interest with that of member states. A similar logic of negotiating domestic political concerns and internal government requirements was noted as a driver of bilateral evaluations by donor countries in the GPPI study.
“All donors interviewed for this review explained that their need for domestic accountability to actors including lawmakers, national oversight bodies or their citizens requires them to seek assurance from their partners about the use of their financial contributions. But there are differences in how such need for domestic accountability enters the donor-agency relationship. Especially factors such as the rise of nationalistic tendencies, skepticism toward the value of humanitarian assistance, shrinking national budgets, consistent scrutiny by government oversight offices, or reviews finding deficiencies in donor oversight might cause donors to tighten their grip on how they conduct oversight vis-à-vis agencies.” - GPPI, Independent Review of Individual Donor Assessments in Humanitarian Operations, Nov. 2020
All of these contexts are highly politicized, yet clearly data and evidence appears to be a useful tool in finding agreements across different political interests and objectives. At the same time, there is a risk that because evaluations are a tool for political negotiations that evaluation outcomes may be manipulated to support these political objectives. Both the GPPI study and Profs. Eckhard and Jankauskas’ study emphasize that they found no evidence of bias in evaluations. Yet Profs. Eckhard and Jankauskas did find that evaluations are often oriented towards the needs of their primary stakeholder, i.e. member states versus UN leadership, for instance. Given their use as a negotiation tool, this is understandable and not necessarily a problem. But with the proliferation of evaluations for different purposes, by different stakeholders within the UN and across member states, there is a risk that evaluations may result in contradictory findings, or results of evaluations could be taken out of context to fulfill a political purpose. Efforts to obtain evaluation results that support a particular political position could also reinforce the risk of creating significant additional - and potential unnecessary - costs for donors and the UN system.
Mitigating the risks
How can these risks and negative consequences be mitigated? One obvious answer which has been highlighted in reform discussions like the 2016 World Humanitarian Summit is to undertake joint assessments and evaluations to reduce costs. The GPPI study found that only 2% of humanitarian donor assessments between 2016 and 2019 were joint assessments, primarily because they were project focused and were driven by domestic considerations and timelines. While this means that joint assessments between donors remains unlikely in the future, it also implies that joint evaluations between UN agencies on policy issues decided by member states could be an area for growth. Joint thematic evaluations across multiple UN entities with similar mandates and governance bodies could lead to political benefits and cost savings. The UN agencies involved could potentially save costs through collaboration, and member state delegates across the different Executive Boards would have one set of data and evidence from which to develop and agree on their policy positions.
Joint assessments will themselves require a certain level of political agreement amongst stakeholders. Therefore another way of mitigating risks and costs is by sharing evaluations and the evidence and data they contain so that they accessible to delegates and practitioners involved in negotiations. An excellent tool in this regard, flagged by one of our readers, is the DAC Evaluation Resource Centre, which is a database of evaluations undertaken by UN entities, multilateral development banks and national governments.(These are the same evaluations also available on individual agencies's websites and presented at UN Executive Board sessions). The evaluations in this database include both country level programmatic evaluations as well as thematic evaluations and therefore provide a wealth of data and evidence for multilateral negotiations on strategic direction and programming for UN governance mechanisms. Self-evaluations and internal assessment exercises for UN leadership can be harder to find, especially for mew leadership teams seeking to understand what works, what doesn't and gather buy-in for their new strategic direction.
Sharing of evaluations and making them accessible is only effective, however, if those negotiating policy positions and strategic direction are aware of the evaluations and the evidence they contain. Both member state delegates and UN senior leadership are inundated with information and data on a daily basis. Therefore, mechanisms are needed to analyze and distill the pertinent elements of evaluation results for them in a way that enhances their negotiation capacity and facilitates consensus-building. This could be an area where UN offices working on learning, training, and/or organizational performance could enhance their contributions and may want to review the extent to which they consider evaluations in their work. Similarly, it could be an area where policy units in ministries of foreign affairs or development could also support their delegates to UN governance mechanisms by synthesizing evidence and data from their own and UN evaluations.
Points for delegates and practitioners to consider: evaluation as a multilateral negotiation tool
Over the next month or two, as discussions on evaluations take place across many UN agencies, funds and programmes, participants preparing their positions and interventions may want to consider the following:
Are there similar UN or member state evaluations in the DAC Evaluation Resource Centre, and how do their findings align with the evaluation currently under consideration by the Board? How does the data and evidence from the evaluation support your message?
To what extent are joint evaluations included in the annual evaluation plan presented to the Board? Considering the objective of the planned evaluations could more joint thematic evaluations with other UN agencies add value? Could such joint evaluations save costs and/or staff time?
コメント