Annex 10: Case Studies of MM Evaluation Designs with Predominant QUANT, QUAL and Balanced Orientations

A. Statistical Designs Using Random Assignment or Quasi-experimental Designs with Matched Comparison Groups

1. Post-conflict reconstruction in Liberia [1]

Project supported by DFID and the International Rescue Committee  

Available at: http://www.alnap.org/pool/files/8rhach2.pdf

Random assignment of communities to receive community driven reconstruction (CRC) treatments (developing community committees and providing support for rapid impact programs). Quantitative social capital questionnaire combined with wide range of QUANT/QUAL data on community organization and social cohesion plus public goods game.

 

2.  Long-term evaluation of the Tostan program to reduce female circumcision in villages in Senegal  

Available at: http://www.unicef.org/evaldatabase/index_59605.html 

The goal of the Tostan (a Senegalese NGO) program was to reduce the prevalence rate of female circumcision, increase age at first marriage and improve the health status of mothers in villages in three regions of Senegal, through promoting social change based on capacity building and participatory development. The long-term evaluation combined a quantitative household survey covering knowledge of female circumcision and prevalence rates, with qualitative techniques to assess program implementation, to understand how villages organized their participation in public declarations, and to obtain women’s opinions about the impact of the program. Three groups of villages were compared: villages that had benefited from a Tostan program and had publicly declared that they would abandon the practice of circumcision; villages that that had made a public declaration to abandon female circumcision but did not benefit directly from a Tostan program; and a control group of villages not exposed to the program.

 

3.  Evaluating a conditional cash transfer program in Kazakstan

Save the Children.

PowerPoint presentation available at: /document/impact-evaluation-save-children

This was the first conditional cash transfer program in a post-Soviet country and the purpose of the evaluation was to assess program impact on the attendance of pregnant women for screening at health clinics and increased attendance of children at pre-school. A mixed method design combined random assignment of villages with two qualitative studies and cost-effectiveness analysis.

 

4. Evaluating the Kecamatan Development Project in Indonesia  

World Bank and Government of Indonesia

Available at: Michael Bamberger, Vijayendra Rao and Michael Woolcock. 2010.  Using Mixed Methods in Monitoring and Evaluation: Experiences from International Development (http://ssrn.com/abstract=1545914)

Sequential mixed method design starting with QUANT survey in two provinces in two regions with high and low levels of conflict. Villages in each region statistically matched using propensity score matching. Followed by 3 months of intensive QUAL work to understand patterns of conflict resolution and to monitor processes of change following project interventions.

 

5.  Evaluating the Indian Panchayat Reform Program

World Bank and Government of India. 

Available at: Michael Bamberger, Vijayendra Rao and Michael Woolcock. 2010.  Using Mixed Methods in Monitoring and Evaluation: Experiences from International Development  (http://ssrn.com/abstract=1545914)

Two contiguous states, one with a high level of community participation and one with a low level, are compared. A sample of panchayats are randomly assigned to the treatment (two weeks participatory planning training) and control groups. Trained observers visited five treatment and five control panchayats for an average of two days per month over a period of two years to observe the processes of community change and organizational development using QUAL methods.

 

6.  Impact evaluation of FAO emergency and rehabilitation work in rural DRC  

Available at: http://www.alnap.org/pool/files/8rhach2.pdf

A number of QUAL methods - including recall, story-telling, focus groups and a qualitative survey of NGOs - were combined with a household survey with matched comparison group. The difficulties of achieving a good match during a period of rapid refugee influx were recognized.

 

B.  Qualitatively Oriented Evaluation Designs

7.  Evaluating the impacts of a gender-violence prevention program in El Salvador

Oxfam America

PowerPoint presentation available at: /document/impact-evaluation%0Bexperiences-oxfam-america  

The program sought to reduce gender-based violence through combining three approaches to promote: an opportunity structure, social relations and behavioral change and well-being and agency. The evaluation was based on the theory of change framework developed as part of the project design and combined: in-depth comparative case studies, major cross-stakeholder document synthesis and interviews on effectiveness of advocacy choices. Attribution analysis was assessed at the outcome level for each project and the awareness campaigns. Contribution analysis was assessed at the broader program level through combining project data, case studies (comparison cases and within case sampling) and tracing influence pathways identified in the theory of change.

 

8. Life and livelihoods food security program in Bangladesh

USAID, Tango International and Save the Children  

Available at: Tango International. 2009. Jibon O Jibika (Life and Livelihoods) Title II Program of USAID in Bangladesh.  Final Evaluation Report (http://pdf.usaid.gov/pdf_docs/PNADS875.pdf)

Quantitative endline survey combined with rapid qualitative methods such as focus groups and key informant interviews.

 

9.  Evaluation of the UNICEF Education Programme in Timor-L’Este (2003-2009)

“From Emergency Responses to Sustainable Development for Children and Adolescents in Timor-L’Este.” 

Available at: http://www.unicef.org/evaldatabase/index_58819.html

This case illustrates how equity issues can be addressed in a context where there is only limited access to quantitative data, and the evaluation must mainly rely on a mixed-method approach.

 

10.  Evaluating the equity-outcomes of the Nepal Education for All Project

Available at: http://www.unicef.org/evaldatabase/index_58884.html

The evaluation did not have a specific equity focus but national partners requested that the sample selection be targeted at some of the poorest and most remote communities, where ethnic minorities and other vulnerable groups represented a high proportion of the population.

 

11.  Evaluating the equity outcomes of the Cambodia Community-Led Total Sanitation Project

Available at: http://www.unicef.org/evaldatabase/index_57963.html

One of the central objectives of the project was to develop methodologies to ensure the participation of all sectors of the population, including the poorest and most vulnerable. A central goal of the evaluation was to assess the equity outcomes of the project.

 

12.  Inter-Agency Real-Time Evaluation of the Humanitarian Response to Pakistan’s 2009 Displacement Crisis

Available at:  http://www.unicef.org/evaldatabase/index_59598.html

This case illustrates how equity issues were addressed in the evaluation of the response by the international community to the humanitarian crisis created by a massive population displacement  in Pakistan. It describes the use of a mixed-method approach that sought to ensure the credibility of the evaluation findings through the presentation of an evidence table and the systematic use of triangulation. It also documents the many political, security and logistical challenges in conducting an evaluation in a military emergency situation. The case illustrates the importance of an equity focus as programs were mainly planned in consultation with village elders and male household heads and little attention was given to the special needs of women and children and the poorest and most vulnerable families.

 

13.  Evaluating the road construction component of the Eritrea Community Development Fund  

World Bank and Eritrean Community Development Fund

Available at: Michael Bamberger, Vijayendra Rao and Michael Woolcock. 2010.  Using Mixed Methods in Monitoring and Evaluation: Experiences from International Development  (http://ssrn.com/abstract=1545914)

Three rural roads were studied during and after the completion of construction. A stratified sample of households in selected villages with access to the raods were interviewed using recall to assess the changes that had taken place in terms of access to schools and clinics as well as agricultural production and sales. These were complemented by key informant interviews, observation, focus groups and review of secondary sources.

 

14.  Evaluation of the Egyptian Community Schools Project

Available at: http://www.unicef.org/evaldatabase/index_59600.html

This case describes an equity-focused evaluation that was specifically designed to assess the effectiveness of community-based schools in increasing school enrollment and performance for under-served population groups, with particular attention to girls. It also discusses the practical challenges of identifying a well-matched comparison group. Both quantitative and qualitative data collection methods were used, but there is no discussion of how these are integrated into a mixed-method strategy or how triangulation is used to strengthen validity of the data, findings and conclusions.

 

15.  Evaluation of the Tanzania Community Justice Facilitation Project  

Available at: http://www.unicef.org/evaldatabase/index_59601.html

This case describes an equity-focused evaluation that assesses the effectiveness of the community justice facilitation project in ensuring that justice is accessible to women and children. It combines quantitative and qualitative data collection methods but does not describe an integrated mixed-method approach or the use of triangulation to strengthen the validity of the data and findings. The practical challenges in conducting a rigorous evaluation design within a multi-level administrative system are also described.

 

16.  Evaluating UNICEF’s response to the 2004 Indian Ocean tsunami in the area of child protection in Indonesia (2005-2008)

Available at: http://www.unicef.org/evaldatabase/index_59604.html

The evaluation, which was commissioned by UNICEF’s Child Protection Department, was aimed at determining the impact of UNICEF's response to the tsunami within the child protection sector, and drawing lessons learned and recommendations for both the recovery/transition and on-going development programming, and policies to improve the well-being and rights of children and women. It follows the evolution of the three child protection work strands (children without family care, psycho-social support, and exploitation and abuse) through the different phases of their development and it examines the extent to which child protection results were achieved in each phase and are likely to be sustained.

Six cross-cutting issues were examined: a) advocacy, policy and coordination; b) reaching the most vulnerable; c) gender; d) conflict; e) emergency, recovery, and early development linkages; and f) child protection systems capacity development.

The evaluation employed a sequential mixed-methods approach to combine comprehensive coverage with in-depth analysis. It focused on three districts to enable comparison of results between tsunami and conflict (mainly) affected districts, which allowed for comparisons between those areas with a strong operational UNICEF presence and those areas with less. The evaluation design also compared different interventions with one another—or, where a similar programme did not exist, with groups of children who did not receive the intervention.

 

C. Meta-analysis [secondary analysis of a number of evaluations to identify general findings]

17.  CARE International. Impact Evaluation Report 2005-2010. Latin America and the Caribbean

Available at: http://www.care.org/newsroom/specialreports/latinamerica/LAC_impact_report-English.pdf

Meta-analysis combining Most Significant Change methodologies with stakeholder surveys.

 


[1] Fearon, J Humphreys, M & Weinstein, J (2009) “Can Development Aid Contribute to Social Cohesion after Civil War?  Evidence from a Field Experiment in Post-Conflict Liberia”.  American Economic Review. Papers and Proceedings. 2009: 99:2 287-291