Measuring Agency-Level Results: Lessons Learned from Catholic Relief Services’ Beneficiary and Service Delivery Indicators Initiative
Main Article Content
Abstract
Background: Many NGOs have less success documenting their results at the agency level than at the program or project level. Little has been published on the challenges NGOs face in developing and measuring agency-level results. To address this issue, InterAction, an alliance of NGOs, commissioned a comparative study that drew on the existing grey literature, and a sample of 17 InterAction member organizations through case studies and interviews.
Purpose: This paper builds on that InterAction study by presenting one of the first published case studies of a successful agency-level measurement (ALM) system – Catholic Relief Services’ (CRS’) Beneficiary and Service Delivery Indicators (BSDI) initiative.
Setting: A faith-based multi-national relief and development NGO.
Intervention: N/A
Research Design: A case study approach was used to describe and document the development of the CRS ALM.
Data Collection and Analysis: The information in this study is derived primarily from CRS files and documents. Data reflecting ALM practices in other NGOs were derived from the 17 InterAction member NGOs. Data reflecting the ALM practices developed by specific NGOs and presented in tabular form in the paper were derived from official documents published by those NGOs.
Findings: The authors discuss key lessons for other large and small organizations to consider when developing their own ALM systems.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Abdul Latif Jameel Poverty Action Lab at the Massachusetts Institute for Technology. (2016). About: History [Web page]. Retrieved from https://www.povertyactionlab.org/about-j-pal
Banerjee, A. & Duflo, E. (2011). Poor economics: A radical rethinking of the way to fight global poverty. New York, NY: PublicAffairs.
CARE International. (2011). Asia impact report—2005 to 2010. [Web page]. Retrieved from http://www.care.org/sites/default/files/documents/Asia-Impact-Report-2010.pdf
Carr, H., & Sharrock, G. (2011). BSDI status report. Catholic Relief Services.
Carr, H. (2011) Initial field visit: Scope of work. Catholic Relief Services.
Carr, H. (2012a). Trip report 2012, April. Catholic Relief Services.
Carr, H. (2012b). Trip report 2012, May. Catholic Relief Services.
Carr, H. (2012c). Trip report 2012, September. Catholic Relief Services.
Carr, H. (2013). Trip report 2013. Catholic Relief Services.
Carr, H. (2014). Trip report 2014, September. Catholic Relief Services.
Carr, H. (2016). Trip report 2016. Catholic Relief Services.
Catholic Relief Services (CRS). (2013a). FY2013 End-of-year project beneficiary reporting.
Catholic Relief Services (CRS). (2013b). Guidance and Support Materials for FY2013 End-of-Year Project Beneficiary Reporting.
Catholic Relief Services (CRS). (2013c). Internal communication between CRS MEAL, finance department, and country programs. [Internal communucation].
Catholic Relief Services (CRS). (2014a). Gateway reports and dashboards job aid. [PowerPoint].
Catholic Relief Services (CRS). (2014b). Gateway frequently asked questions (FAQ).
Catholic Relief Services (CRS). (2015). Monitoring, evaluation, accountability and learning (MEAL) policies & procedures. [Document # POL-OOD-PRG-008].
Charity Navigator (2016). [Web page]. Mission, vision, and goals. Retrieved from http://www.charitynavigator.org/index.cfm?bay=content.view&cpid=17
Food Assistance and Nutrition Technical Assistance (FANTA). (1999, January). Food security indicators and framework for use in the monitoring and evaluation of food aid programs. [pdf]. Retrieved from http://pdf.usaid.gov/pdf_docs/Pnacg170.pdf
Global Partnership for Effective Development Co-operation. (2016). About the partnership. [Web page]. Retrieved from http://effectivecooperation.org/about/
Hagens, C. (2011, November). Discussion notes, field validation of BSDI in Asia region. Catholic Relief Services.
Hagens, C. (2012). Trip report 2012, May. Catholic Relief Services.
Hughes, K. & Hutchings, C. (2011). Can we obtain the required rigour without randomisation? Oxfam GB's non-experimental global performance framework. [pdf]. Retrieved from Internation Program for Development Evaluation Training website: http://www.ipdet.org/files/Publication-Can_we_obtain_the_required_rigour_without_randomization.pdf
International Aid Transparency Initiative. (2016). About IATI. [Web page]. Retrieved from http://www.aidtransparency.net/about
International Initiative for Impact Evaluation (3ie). (2016). About 3ie [Web page]. Retrieved from http://www.3ieimpact.org/en/about/
International Federation of Red Cross and Red Crescent Societies. (IFRC). (2013). Federation-wide data bank and reporting system (FDRS) user guide 2013.
Leeuw, F. & Vaessen, J. (2009). Impact evaluations and development: NONIE guidance on impact evaluation. Retrieved from Active Learning Network for Accountability and Performance in Humanitarian Action website: http://www.alnap.org/resource/8194
Levine, C.J., van Vijefeijke, B., & Jayawickrama, S. (2016). Measuring international NGO agency-level results [White paper]. Retrieved from InterAction website: https://www.interaction.org/document/measuring-ingo-agency-level-results
Lucas, S.T. (2011). Principles into practice: Focus on results. [pdf]. Retrieved from Millennium Challenge Corporation website: https://assets.mcc.gov/reports/paper-2011001052001-principles-results.pdf
Lutheran World Relief (LWR). (2015). Lasting impact: 2014 annual report. Retrieved from http://lwr.org/wp-content/uploads/ANNUAL-REPORT-2014.pdf
Manning, R. & White, H. (2014). Measuring results in development: The role of impact evaluation, in agency-wide performance measurement systems, Journal of Development Effectiveness, 6(4), 337-349. https://doi.org/10.1080/19439342.2014.989673 DOI: https://doi.org/10.1080/19439342.2014.989673
Mercy Corps. (2014). Mission metrics: Real data for meaningful change – 2013 report. [pdf]. Retrieved from http://www.mercycorps.org/sites/default/files/Mission%20Metrics%202013%20Report.pdf
NetHope. (2016, July 20). Our approach. [Web page]. Retrieved from http://nethope.org/our-approach/
Organization for Economic Co-operation and Development's Development Assistance Committee (DAC). (2016). The high level fora on aid effectiveness: A history [Web page]. Retrieved from http://www.oecd.org/dac/effectiveness/thehighlevelforaonaideffectivenessahistory.htm
Organization for Economic Co-operation and Development's Development Assistance Committee (DAC). (n.d.). Paris Declaration on Aid Effectiveness: Five principles for smart aid [Brochure]. Retrieved from http://www.oecd.org/dac/effectiveness/45827300.pdf
Pact. (2012). Measuring pact's mission – 2012. [pdf] Retrieved from http://www.pactworld.org/sites/default/files/2012MeauringPactsMission_Pt2_0_0.pdf
PEPFAR. (2013). Next generation indicators reference guide version 1.2. Retrieved from http://www.pepfar.gov/documents/organization/206097.pdf
Rugh, J., Pechi, L., & Nkwake, A. (2008). Transformational development indicators: Summary report from the TDI evaluation team. Unpublished internal report, World Vision.
Save the Children. (2013). Save the Children thematic-specific guidance for estimating total reach.
Savedoff, W.D., Levine, R., & Birdsall, N. (2006). When will we ever learn: Improving lives through impact evaluation. [pdf] Retrieved from Center for Global Development website: http://www.cgdev.org/files/7973_file_WillWeEverLearn.pdf
United Nations Programme on HIV/AIDS. (2011, October). Global AIDS response progress reporting 2012. Retrieved from http://files.unaids.org/en/media/unaids/contentassets/documents/document/2011/JC2215_Global_AIDS_Response_Progress_Reporting_en.pdf
USAID. (2008). Foreign assistance master list of standard indicators. [pdf] Retrieved from U.S. State Department Archive website: http://2001-2009.state.gov/documents/organization/115255.pdf
USAID. (2011, January). USAID announces USAID FORWARD reform agenda. [speech] Retrieved from https://www.usaid.gov/newsinformation/speeches/remarks-usaid-administrator-dr-rajiv-shah-center-global-development
USAID, Bureau of Policy Planning and Learning. (2011). USAID evaluation policy. [pdf] Retrieved from https://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf
United States Department of Agriculture, Foreign Agricultural Service (FAS), Office of Capacity Building and Development. (2013). Monitoring and evaluation policy. Retrieved from http://www.fas.usda.gov/sites/default/files/201403/evalpol.pdf
World Bank, Development Impact Evaluation (DIME). (2016). Development impact evaluation [Brochure]. Retrieved from http://pubdocs.worldbank.org/pubdocs/publicdoc/2015/11/728271447880371443/DIMEBrochureRed.pdf