2021 in review
- Completed 13 independent evaluations
- Initiated 23 new evaluations, each aligned with priority areas of UNHCR’s work – an increasing number being commissioned by Regional Bureaus and Country Offices
- Developed a strategic approach to evaluate UNHCR’s COVID-19 response
- Completed United Nations Evaluation Group-DAC peer review of the evaluation function. Received valuable feedback that has informed new policy
- Expenditure on evaluation across the organization is estimated at $ 7.5 million USD in 2021 or just over 0.1% of overall global programme expenditure
- Produced new guidance to support a more efficient workflow with regards to the quality and timeliness of the management responses
- Global thematic evaluations directly feed into new focus area strategies under UNHCR’s Strategic Directions 2022-2026
Introduction
Over the past two years, COVID-19 required an unprecedented degree of adaptation and agility from the Evaluation Service, much as from the rest of the UNHCR.
We have adjusted our methods of evaluation to overcome the new challenges. We devised remote mission strategies, innovative data collection methods, and found new ways of sharing information.
In 2021, we completed 13 independent evaluative exercises and initiated another 23 - from operationally specific to global thematic - each aligned with the organization’s strategic priority areas.
We would like to share with you the progress we have made towards achieving our strategic goals as set out in the 2018-2022 Evaluation Strategy:
- Increasing evaluation coverage and quality
- Improving the relevance and utilization of evaluation findings
- Building capacity to undertake and use evaluations
- Strengthening an evaluation function that is linked to other complementary functions, such as oversight, strategic planning, monitoring, data and analytics, and results-based management.
The number of evaluations has grown steadily since the Evaluation Policy was approved in 2016 with only two centralized and two decentralized evaluations completed that year. The Evaluation Workplan for 2021 included 37 evaluations (15 centralized, 5 inter-agency, and 17 decentralized evaluations).
An external facility reviewed and assessed all UNHCR evaluations completed in 2021 and gave a quality rating of 71% - compared to 58% in 2018/2019 baseline assessment. The independent quality assurance system initiated in 2018/19 and rolled out to all evaluations by 2021 is being used by evaluation managers at HQ, divisions, bureaux and country operations to ensure compliance with international evaluation norms and standards.
As part of the roll-out of multi-year strategic planning, a relatively new type of evaluation has been added to management decision-making tools: the Country Strategy Evaluation (CSE). These evaluations take a holistic look at UNHCR’s results at a country level and examine issues of partnership, comparative advantage and the mix of tactics used to achieve results. There has been a significant increase in the demand for this type of evaluation – going forward we expect to undertake 1-2 CSEs per region per year.
Looking forward and more generally, we see significant growth in demand for and coverage of evaluations commissioned at the regional and country-level, stimulated by regionalization and decentralization and the roll-out of strategic multi-year planning at all levels.
OVERVIEW OF EVALUATIONS COMPLETED IN 2021
Decentralized evaluations completed in 2021
- Caring for refugees with non-communicable diseases project
- UNHCR’s Child Protection Programming 2017-2019
- Somos Panas Colombia Communication Campaign
- “Saving Maternal and Newborn Lives in Refugee Situations” in Cameroon, Chad and Niger
- Mid-Term Process Evaluation of the IKEA Foundation Livelihoods and Energy Projects Among Somali Refugees and Host Communities in Ethiopia
- Action Access pilot
- Telling the Real Story Project 2.0
- UNHCR/UNICEF Fair Deal (Blueprint) for Refugee Children Round 1 (first-year interim report - Internal)
Other evaluative briefs and reports produced in 2021
- COVID-19 Evaluative Evidence Brief No. 1
- COVID-19 Evaluative Evidence Brief No. 2
- Peer Review of UNHCR’s evaluation function
- Independent Race Equality and Equity Review (Internal)
Evaluations become relevant when they focus on topics that resonate with users' interests; when the findings are communicated effectively in a timely manner, and when the results prompt conversations and new thinking. We have made progress on all of these aspects this past year.
Evaluative work is informing UNHCR’s thinking and programming, in particular our new Strategic Directions 2022-2026, including:
- New climate action plans at global and regional levels and revisions to the existing emergency response policy (informed by lessons from the evaluation of UNHCR’s 2019/20 Cyclone Idai response in Southern Africa);
- Major shifts in how the organization delivers on learning beyond training - Global Learning and Development Center Transformation 2022 (informed by the evaluation of workforce learning and development);
- New organizational emphasis on UNHCR’s mandate on statelessness (underpinned by the evaluation of UNHCR’s efforts to end Statelessness);
- Operational scale-up in northern Ethiopia following the recent L3 declaration (informed by the internal real-time review).
- New global focus area strategy on Engaging with Development action (informed by the longitudinal evaluation of UNHCR’s engagement with Humanitarian-Development Cooperation).
A comprehensive communication strategy, developed in 2020, is now being rolled out. The Evaluation Service has always approached the conduct of evaluations in a highly participatory way, and this is being complemented by new approaches to reach new audiences and to socialize evaluation findings using short briefs, videos, infographics, and interactive web-based evaluation content. A good example of these new approaches is communicating the results of the Humanitarian Development Cooperation evaluation, which included a comprehensive communications campaign, multimedia products, digital launch, and presentation events. The campaign reached over 1800 people internally and externally, with the report being downloaded over 960 times.
Another good example is the communications around a decentralized Evaluation of the Action Access pilot. As part of the dissemination efforts, the team produced and disseminated a press release on the Report. As a result, the news made headlines in the BBC, The Guardian and other media in the UK, stimulating public discussion about the report and its findings.
UNHCR is committed to the timely uptake and use of evaluation recommendations, and we are pursuing three specific actions to support it. First, expanding the use of strategic workshops that focus specifically on fine-tuning recommendations to ensure that they are useful and actionable. Second, introducing a ‘light touch’ follow-up system designed to track the timeliness of submission and implementation of the evaluation management responses. Three, greater efforts are being made to capture qualitatively the impact of recommendations on decision-making.
In this short video, UNHCR’s Assistant High Commissioner for Protection, Gillian Triggs shares how evaluative evidence produced by the Evaluation Service is helping the organization to deliver on its protection mandate and beyond.
We are committed to strengthening internal capacity to commission and use evaluations to improve the organization's work. We have finalized step-by-step guidance for the conduct of decentralized evaluations and established two new posts in the East and Horn of Africa and the Great Lakes and West and Central Africa Regional Bureaux – with a third on the way in the Middle East and Northern Africa region. These regional positions are embedded and in particular, will support the development of the country and regional evaluations plans – based on identified needs for learning and strategic planning inputs.
Regional budgets for evaluation linked to regional evaluation plans have been prepared through a cost-sharing arrangement between the Bureaux and the Evaluation Service – all with the view to stimulating evaluation demand and ensuring that evaluation contributes to robust management in the context of devolving decision-making and multi-year planning.
In terms of evaluation competency development, on-the-job learning and coaching by the Evaluation Services' evaluation managers in connection with the implementation of evaluations remained a core approach. Alongside experiential learning, we implemented in-the-flow-of-work agile learning approaches such as, inter-alia, the establishment of a virtual Monitoring & Evaluation Community of Practice, in partnership with the Division of Strategic Planning and Results; delivering an experience sharing session on managing an evaluation at the global ‘Learn@work’ week in collaboration with the Global Learning and Development Center; and issuing a modular, audience-sensitive ES Onboarding Guide to ensure a smooth and structured induction into UNHCR’s evaluation function.
With regards to formal training, we actively supported the ‘UN as ONE’ approach by engaging with the UN Evaluation Group (UNEG) on the development of two key initiatives aligned with the UNEG Evaluation Competency Framework and planned to be launched in 2022: (a) UN system-wide certificated course for evaluators and evaluation managers at an intermediate level, and (b) establishment of a UNEG evaluation section within the SDG Learn portal, featuring a curated list of a wide range of learning opportunities, from self-paced micro-learning to full-fledged learning programmes.
During the 2020-21 roll-out of the new Results-Based Management (RBM) system COMPASS, we have provided continuous support –including resource persons for training as part of multi-functional teams, contributing to strategy development, reviewing the new RBM guidance, supporting regional and country-level monitoring and evaluation plan development, and responding to questions of operational colleagues.
Evaluation is working in synergy with other oversight providers within UNHCR. Under the leadership of the Inspector General’s Office, efforts are underway to coordinate audit and evaluation activities with the view to a) providing a holistic view on organizational performance from the activity level to results for persons of concern, and b) optimizing coverage while paying attention to the ‘footprint’ such activities have on operations.
To improve our understanding of UNHCR positioning and results in the broader ecosystem of humanitarian partners working on the ground, we are engaged in several inter-agency and system-wide evaluations including a joint evaluation of The UNHCR/UNICEF BluePrint for Refugee Children, an evaluation of the overall humanitarian response in Yemen (OCHA led) and a global interagency evaluation that examined the collective performance of humanitarian actors in addressing gender equality and the empowerment of women and girls.
The Head of Evaluation, UNHCR is Vice-Chair of the UNEG. The Evaluation Service is an active participant of UNEG, engaging in a variety of themes and groups, including Peer-Review, Ethics, Gender, Disability, Human Rights, Professionalization, Evaluation Policy Influence, and other working groups, and was represented on the panel of World Food Programme’s Evaluation Function Peer Review.
Findings from the selected evaluations
Cyclone Idai Response
The evaluation of UNHCR’s Level 3 emergency response to Cyclone Idai in Mozambique, Malawi and Zimbabwe concluded that while UNHCR played a critical role in coordinating and leading protection efforts, there were critical ‘policy-implementation gaps’ concerning the organization’s engagement in climate-related disaster displacement in non-conflict situations. The study highlighted the need for better policy coherence to guide UNHCR responses to disasters and has been catalytic in prompting management action with regard to disaster preparedness planning. It identified lessons that can be extrapolated to other disaster responses and provided valuable inputs into UNHCR’s ongoing reflection on climate change and disaster-induced displacement.
In this video, we summarized the Report's key findings:
UNHCR-led Initiatives to End Statelessness
The evaluation found that the Global Action Plan (GAP) has been a useful framework for guiding States, UNHCR, and other partners. However, the ambitions of the campaign and targets set within the GAP are unlikely to be achieved by 2024. While UNHCR has been particularly successful in implementing initiatives to address the legal and administrative causes of statelessness, other key drivers of statelessness, including political, social, and economic aspects, need still to be addressed.
Gaps in reliable data on stateless persons and the effect that statelessness has on their lives have made it more difficult to communicate and prompt action. Internal leadership on statelessness has been crucial to successes so far with Country Representatives playing the most critical role of all. However, UNHCR's statelessness budgets have not kept pace with its ambitions. Therefore, the evaluation argues that additional, sustained funding (and fundraising) is required. Equally crucial is an investment in UNHCR´s workforce, to build knowledge and skills, with more dedicated staff and communications capacity. UNHCR has successfully mobilized a wide range of stakeholders at different levels and the evaluation recommends further efforts to bring international, regional, and national actors together as a true coalition to collectively mobilize for change, including harnessing the contributions of stateless persons directly.
Egypt Country Strategy 2016-2020
The evaluation found that UNHCR is widely credited with contributing to the national asylum law, which is a potentially seminal achievement. Excellent relations with the Government of Egypt have been a key ingredient to this success. UNHCR holds a unique role in protection services and has achieved significant successes with comparatively limited funding in Egypt. However, there are other actors who can – and should – provide essential sectoral services to meet basic needs, and, by influencing them to increase investments, more of UNHCR’s budget could be allocated to the protection services that no other agency can provide.
The evaluation recommended that UNHCR Egypt should consider how best to continue working closely with the government while ensuring that the organization maintains its protection authority including for the safeguarding of the rights of refugees; that UNHCR develops an approach that is more catalytic to encourage greater burden-sharing with other partners; and, that the agency review and revise their approach to Communicating with Communities.
UNHCR’s Engagement with Humanitarian-Development Cooperation
This longitudinal evaluation, launched in 2018, aims to help UNHCR position itself and approach its cooperation with development partners and host governments to support refugee self-reliance and inclusion.
The evaluation has found that UNHCR has implemented a number of relevant, effective institutional measures to support the inclusion agenda, including creating institutional units and staff positions to drive engagement with development actors. The four Discussion Papers published late in 2020 focussed on closing policy-operational gaps, planning and budgeting processes that support the GCR, strengthening UNHCR's engagement in systems building for protection, and further opportunities for strengthening bilateral and UN development system partnerships towards the inclusion of refugees.
This short video summarizes the evaluation's key findings and recommendations:
"Action Access", an Alternative to Detention Pilot (UK)
Decentralized evaluation
Since 2015, the number of people being detained in the UK has gradually declined and several Immigration Removal Centres have closed. However, the UK still has one of the largest detention estates in comparison with European countries and, unlike EU countries, has no time limit on immigration detention. One of the recommendations of the Shaw Progress Report, published in 2018, was that the Home Office establish an Alternatives to Detention (ATD) project. The Detention Reform Program, started in 2018, set out a strategic direction for reducing the use of immigration detention in the UK and a wide range of reforms to underpin that including developing ATD pilots.
In response and after working closely with UNHCR, the UK government announced the Community Engagement Pilot (CEP) Series. The overall principle of the CEP Series is to test approaches to supporting people to resolve their immigration case in the community.
UNHCR commissioned NatCen Social Research to undertake an independent evaluation of Action Access, the first pilot in the CEP series. Delivered by Action Foundation over two years from 2019 to 2021, Action Access aimed to support women with asylum-seeking status in a community-based, engagement-focused ATD through the provision of one-to-one support from a support worker, shared accommodation, and legal counseling from a qualified legal professional.
The evaluation found that it is more humane and significantly less expensive to support women in vulnerable situations in the community as an alternative to keeping them in detention centres.
COVID-19
Evaluative Synthesis of UNHCR’s adaptation and response to the COVID-19 pandemic
The purpose of the synthesis is to provide robust and timely evidence to UNHCR on how effectively the organisation has adapted to COVID-19 and to highlight areas of strength as well as the challenges emerging from across our evaluations.
Amongst a range of effects resulting from or exacerbated by COVID-19, evaluations find that Persons of Concern (PoC) are facing increased economic hardships, as livelihood opportunities are constrained and poverty rates increase. Evaluations also note increased reports of sexual and gender-based violence (SGBV) and delayed access to services, and increased xenophobia and discrimination. Evaluations also signal reduced access to basic services.
Evaluations find that UNHCR has adapted its work to respond to COVID-19 challenges. These include maintaining contact with PoC through remote methods such as telephone helplines, partners and community contacts, reallocating budgets and more.
Evaluations analyzed found UNHCR’s response to be relevant overall, with some challenges, such as difficulties in directly accessing PoC, reduced attention to issues such as statelessness, and restricted capacity development and livelihoods approaches which are essential for tackling the socioeconomic effects of the crisis.
The Protection of Refugees during The COVID-19 Pandemic: Joint Evaluation under the Auspices of the Global Evaluation Coalition
The purpose of this joint evaluation is to examine the effectiveness of international cooperation and the combined response of key actors in ensuring the protection of the rights of refugees during the COVID-19 pandemic: to identify emerging good practices, innovation and adaptation to protection responses.
The joint Evaluation is carried out under the auspices of the Global Evaluation Coalition, facilitated by OECD DAC Evaluation Network (EvalNet). The Management Group for this evaluation includes the Evaluation Units of UNHCR, the Ministry of Foreign Affairs of Finland, the Governments of Colombia and Uganda, and the humanitarian system network ALNAP, and chaired by UNHCR.
Emerging findings from the evaluation were presented at the High-level Official Meeting’s spotlight session on the medium-long term effects of the COVID-19 pandemic.
The report is planned to be finalized in May 2022, followed by a series of global and regional launch events, where key findings and conclusions will be discussed.
In this video, Anila Noor, a refugee activist and a member of the reference group for the COVID-19 joint evaluation shares her thoughts on why evaluations are crucial and how they can help refugees and other persons of concern
The Year Ahead
In 2022, we will face new challenges in light of the rapidly evolving humanitarian crisis in Ukraine as well as ongoing major emergencies in, inter alia, the DRC, the Sahel, Afghanistan, Ethiopia.
The Evaluation Service is supporting UNHCR by evaluating major emergency responses in all of these major emergency settings.
We are completing important strategic evaluations that look at UNHCR support for voluntary repatriation and reintegration as well as at UNHCR’s efforts to build national asylum capacity. Our year 1 report on the implementation of UNHCRs policy on Age, Gender and Diversity is very soon to be published.
In addition, we plan to start the following evaluations in 2022:
- Country Strategy Evaluation: Ecuador
- Country Strategy Evaluation: Nepal
- Country Strategy Evaluation: South Sudan
- Country Strategy Evaluation: Uganda
- Country Strategy Evaluation: Zimbabwe
- Engagement in Situations of Internal Displacement
- Programming to Prevent Gender-Based Violence
Selected evaluations to be commissioned by Divisions, Regional Bureaus and Country Offices:
- Costa Rica Basic Needs and Livelihoods Evaluation
- End line Evaluation Livelihoods and Energy Projects among Somali Refugees and Host Communities in Ethiopia
- Evaluation of UNHCRs Livelihood Programme in Syria
- Humanitarian Education Accelerator
- Implementation of the Emergency Transit Mechanism
- Livelihoods Programmes in Algeria
- Multi-Country Livelihood Programming in Asia
- Multi-Country R-WASH Programme in the Horn of Africa
In 2022, a new evaluation policy will be developed and approved, drawing significantly from the 2021 UNEG/DAC Professional Peer Review of UNHCR’s Evaluation Function. Building on the strengths of the 2016 policy, UNHCR's forward-looking vision for evaluation reflects organization-wide reforms related to regionalization and decentralization as well results-based-management where evaluation is used as a management tool for strengthening learning, oversight and accountability at all levels of the organization.
A message from the Head of the Evaluation Service, Lori Bell
Going forward in 2022, our evaluation planning and practice will align with the Strategic Directions 2022-2026 and its global results framework – seeking to independently validate good practices, prompt course correction where needed, inform decision-making at all levels of the organization, and provide external assurance to member states, partners and persons of concern alike. We want to be vision-and-values-driven, allowing for strong internal and external accountability while remaining responsive to the key questions that keep management awake at night. Most importantly we will be real-time and solutions oriented in order to stay relevant in the fast moving humanitarian contexts within which UNHCR works.
Want to get in touch with us? Contact us at hqevaser@unhcr.org