Relating Outputs, Outcomes and Impact in the Evaluation of International Development Projects
Main Article Content
Abstract
Background: The recent controversies on the pages of the JMDE regarding UNDP evaluations, as well as the DAC criteria, are discussed in the light of two UNDP/GEF evaluations in Latin America for which the author was primarily responsible.
Purpose: The author defends the utilization of all five evaluation criteria of the Development Assistance Committee of the OECD (the DAC criteria), and their integration via Theories of Change.
Setting: The pine/oak forest of Honduras and the mangrove regions along the coast of Brazil.
Intervention: Two projects of the Global Environment Facility (GEF) and the United Nations Development Programme (UNDP), with the respective governments.
Research Design: Mid-term evaluations with site visits and interviews.
Data Collection and Analysis: In the Brazilian case, longitudinal analysis of available data was conducted and related to the findings of the interviews and observation, as well as published reports and studies. A Theory of Change (action model) of the Honduran project was structured and graphically portrayed based on desk review of the project document and other documentation, adapted following initial interviews, and field tested.
Findings: In Brazil, preliminary evidence derived in part from Management Effectiveness Tracking Tool (METT) questions with reference to results indicates a relative lack of project effectiveness and of preliminary signs of impact. The state of Pará offers one possible exception, related in part to the fact that a certain momentum had already been built up in that state prior to project start-up and in part to the early adoption through the project of a new method of transportation of the fiddler crab (Ucides cordatus), in baskets covered with wet sponges rather than in sacks. The Honduran project, with the strong support of the national government and the UNDP field office as well as GEF, successfully adapted the provisions of the project document to pursue a more community- and community organization-centered approach, rather than relying on the outside consultants originally specified. Project-supported gathering of impact data raised environmental awareness, strengthened the local university and established a baseline for future ex-post impact evaluation.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Actknowledge (2011). Introduction to TOC online. Retrieved from http://www.theoryofchange.org/wp-content/uploads/toco_library/pdf/2011_-_ActKnowledge_-_Theory_of_Change_Online_-_overview.pdf
Belokurov, A. (2007). Management Effectiveness Tracking Tool: Reporting progress at protected area sites. Second edition. Gland, Switzerland, WWF International. Center for Theory of Change. Center for Theory of Change. Retrieved from www.theoryofchange.org/toco-software/
Brann, J. and Zhakupova, A. (2012). Steppe conservation and management, Kazakhstan. PIMS 3835. GEF Agency: United Nations Development Programme. Executing Agency: Ministry of Agriculture: Committee on Forestry and Hunting. Astana, UNDP. Retrieved from https://www.google.com.br/?gws_rd=ssl#q=Brann+Zhakupova%2C+2012
Chen, H. (2005). Practical program evaluation: Assessing and improving planning, implementation, and effectiveness. Thousand Oaks, CA: Sage. https://doi.org/10.4135/9781412985444 DOI: https://doi.org/10.4135/9781412985444
Chianca, T. (2008). The OECD/DAC Criteria for international development evaluations: An assessment and ideas for improvement. Journal of Multidisciplinary Evaluation 5(9), 41-45. Retrieved from http://journals.sfu.ca/jmde/index.php/jmde_1/issue/view/25. https://doi.org/10.56645/jmde.v5i9.167 DOI: https://doi.org/10.56645/jmde.v5i9.167
Donaldson, S. I. (2003). Theory-driven program evaluation in the new millennium. In S. I. Donaldson & M. Scriven (Eds.), Evaluating social programs and problems: Visions for the new millennium (pp. 111-142). Mahwah, NJ: Lawrence Erlbaum Associates. https://doi.org/10.4324/9781410606556 DOI: https://doi.org/10.4324/9781410606556
Donaldson, S. I. (2007). Program theory-driven evaluation science: Strategies and applications. New York, Taylor & Francis Group - Psychology Press.
Etzioni, A. (1964). Modern organizations. Englewood Cliffs, NJ: Prentice Hall. Retrieved from http://biblsrc.btk.ppke.hu/Szociologia/08EtzioniAmitai_Modern_Organizations.pdf.
GEF Evaluation Office (2009). The ROtI handbook: Towards enhancing the impacts of environmental projects. Global Environment Facility and Conservation Development Centre. Retrieved from https://www.thegef.org/gef/sites/thegef.org/files/documents/M2_ROtI%20Handbook.pdf
Kinzo, D., Walker, R. and Borges, V-L (2013). BRA/07/G32, Conservação e Uso Sustentável Efetivos de Ecossistemas Manguezais no Brasil, PIMS 3280 - Projeto do Atlas 000559992, Avaliação de Meio Termo. Brasilia, UNDP.
Lempert, D. H. (2010). Why government and non-governmental policies and projects fail despite 'evaluations': An indicator to measure whether evaluation systems incorporate the rules of good governance. Journal of MultiDisciplinary Evaluation, 6(13), 58-108. Retrieved from http://journals.sfu.ca/jmde/index.php/jmde_1/article/view/243 https://doi.org/10.56645/jmde.v6i13.243 DOI: https://doi.org/10.56645/jmde.v6i13.243
OECD (2014). Evaluation of development programmes: DAC criteria for evaluating development assistance. Retrieved from http://www.oecd.org/development/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm
Riemer, M. & Bickman, L. (2011). Using program theory to link social psychology and program evaluation. In M. Mark, S. Donaldson and B. Campbell, Social psychology and evaluation. New York: The Guilford Press, 2011, pp. 104-138.
Roduner, D., Schläppi, W., and Egli, W. (2008). Logical Framework Approach and Outcome Mapping: A constructive attempt of synthesis. Zurich, AGRIDEA and NADEL, ETH. Retrieved from http://www.outcomemapping.ca/forum/files/Discussion_Paper_OM_LFA_Synthesis_2008-1_126.pdf
Senge, P. (1990). The fifth discipline: The art & practice of the learning organization. New York: Currency Doubleday.
Uitto, J.I. (2010). Credibility and independence of evaluation in UNDP: A response to David Lempert. Journal of MultiDisciplinary Evaluation, 6(14), 153-158. Retrieved from http://journals.sfu.ca/jmde/index.php/jmde_1/article/view/285 https://doi.org/10.56645/jmde.v6i14.285 DOI: https://doi.org/10.56645/jmde.v6i14.285
UNDP (2009). Handbook on planning, monitoring and evaluating for development results. New York, United Nations Development Programme.
UNDP (n.d. a). Direct Implementation (DIM) Modality. New York, United Nations Development Programme. Retrieved from https://info.undp.org/global/popp/frm/pages/direct-implementation-dim-modality.aspx
UNDP (n.d. b). National implementation (NIM) finances and NGO implementation finances. New York, United Nations Development Programme. Retrieved from https://info.undp.org/global/popp/frm/pages/national-implementation-nim-Finances.aspx
Walker, R. (2000). Impacting social problems: Writing and evaluating international development projects. Brasilia, EMAD.
Walker, R. (2014). Mid-term review of the project, Mainstreaming Biodiversity Conservation into the Management of Pine/Oak Forests. PIMS 4210. Tegucigalpa, UNDP.