28 May 2023
By Katja Hemmerich
This week, the UNFPA evaluation function informally briefs its Executive Board on its 2022 annual evaluation report and the initial work to update its evaluation policy, which will be presented to the Board for approval in 2024.
On 18 May, UNICEF did the same with their Board, presenting the results of the peer review of their evaluation policy and the new draft, which is expected to be adopted in September 2023. Accordingly, our spotlight this week is on what makes a good evaluation policy, based on a recent exploratory study of 52 evaluation policies from bilateral and multilateral aid agencies.
The study, undertaken by the University of Ottawa, is premised on the assumption that an evaluation policy is intended to facilitate sound evaluation practices across
an organization, and therefore must also build evaluation capacity across the organization. Evaluation capacity is comprised of two important elements: the capacity to do evaluations, and the capacity to use evaluations. Both must go hand in hand.
Both the UNFPA and UNICEF annual evaluation reports capture and measure the importance of capacity to do and to use evaluations. In terms of capacity to do evaluations, both organizations monitor number of evaluations undertaken over time and levels of resourcing to the evaluation office. Because they both decentralize country programme evaluations to the country office, they also track resourcing and skill levels for evaluation in country offices.
In terms of capacity to use evaluations, both organizations track the submission and timeliness of management responses, as a recognition of management’s commitment to implement the recommendations. UNFPA goes further and tracks the extent to which all the management responses were implemented over time and the extent to which evaluations have been used to develop country programming:
"In 2022, the interdivisional Programme Review Committee noted that all new country programme documents (100 per cent) submitted to the Executive Board for approval were clearly informed by evaluative evidence. This marks a significant improvement over 2018, when 78.9 per cent of country programme documents met this requirement." – UNFPA Annual Evaluation Report, para. 36
As part of its policy review process, UNICEF, like UNFPA, initiated a peer review process by the UN Evaluation Group (UNEG). The UNICEF peer review report, issued in April, also highlights the importance of doing and using evaluations, noting for instance that:
"Management responses to evaluation are nearly always produced, which is positive, but a gap remains in evidencing that evaluations have been used to improve policy, programming, practice and outcomes for children." – UNEG Peer Review of UNICEF's evaluation function, pg. vii
The need to use evaluation findings is easy to do on paper, but less so in practice. Thus, evaluation policies need to do more than just reference this as part of the process. Research and practice demonstrates that highlighting the learning objective of an evaluation is an effective way of strengthening stakeholder buy-in, and therefore usage of the evaluation results. Policies that outline collaborative processes for planning, undertaking and sharing evaluation results tended to demonstrate greater buy-in and therefore usage of the evaluation findings in future programming.
As organizations enhance their evaluation experience an interesting tension arises in relation to the stated aim of evaluation policies. The researchers at the University of Ottawa found that in over half of the first iteration of evaluation policies, external accountability demands were the primary drivers of the evaluation policy:
"Over half of the evaluation policies identified the promotion of accountability as the principal goal, often the main reason why the policy was established in the first place. Accountability was often framed in terms of value for money and the demonstration of aid impact." – 'Evaluation Policy and Organizational Evaluation Capacity: A Study of International Aid Agency Evaluation Policies', pg. 36
Over time, however, policy updates increasingly emphasized organizational learning as the principal goal of evaluation. Accountability remained a core objective but it wasn't necessarily the primary one, which seems to reflect a recognition in the organization of the importance of organizational learning in undertaking evaluations with stakeholder buy-in and to strengthen the use of evaluation results.
The problem noted by the researchers is that virtually none of the policies spell out what is meant by learning or accountability, making it difficult to monitor and achieve results in this regard. Defining what is meant by organizational learning helps create more explicit linkages with both the doing and using elements of building evaluation capacity.
Defining accountability also provides an opportunity to bridge some of the accountability gaps inherent in international organizations. International organizations are set up by member states for member states, and accordingly, the accountability of international organizations is to those member states. While positive impact for beneficiaries is a stated aim of international organizations, very few international organizations have any formal mechanisms or processes to hold them accountable to those beneficiaries. (See our primer on accountability or download our UN accountability mechanism infographic for more info.)
Because governance and accountability mechanisms are taking an increased interest in evaluations and receive regular reports on them, it creates an opportunity for member states and international organizations to collaboratively consider impact of programmes, organizations and the money used to fund them, with beneficiaries at the center of the discussion. Framing accountability and learning accordingly in updated evaluation policies can institutionalize and guide such discussions, which can help align a shared way forward to achieving the SDGs.
The current policy updates provide opportunities for UNICEF, UNFPA and others to consider different options for defining accountability with beneficiaries in mind, in addition to member states. One example to consider is WFP's inclusion of host country evaluators and participants or other beneficiaries. WFP has even invited some of these to participate in the recent Evaluation Roundtable discussions with its Executive Board.
The evaluation policy updates therefore are more than just an administrative exercise, they have the power to foster new perspectives on the performance of international organizations. Accordingly, for those involved in developing and updating evaluation policies in their organizations, we suggest considering the following questions:
What is the purpose of the evaluation policy?
How does it define accountability and learning?
Does the policy consider how to do evaluations and how to use the results?
Is impact on beneficiaries captured systematically in evaluation results, and included in reports to governing bodies?
How do beneficiaries participate in the evaluation process?
Comments