Utilization Focused Developmental Evaluation: Learning Through Practice
Main Article Content
Abstract
Background: Utilization-focused evaluation provides an overall decision-making framework with the intention of ensuring evaluation products and processes are actually used. Developmental evaluation provides a structure to learn from an experiment or pilot in the making and provide feedback to course-correct and improve the ongoing effort. In this paper we report on a project where we combined both into a utilization-focused developmental evaluation (UFDE).
Purpose: To determine the theoretical complementarities and the practical advantages of combining UFE with DE, by reflecting on a practical experience. We include a synopsis of the methodology along with a sample of findings, followed by a reflection of the overall process. We emphasize the conditions that enabled this experience to evolve to guide other practitioners interested in this learning approach to evaluation.
Setting: The context was the piloting of a social and financial education curriculum for youth called Aflateen that was developed by Aflatoun Child Savings International in Amsterdam and test-driven by over forty partners around the World.
Intervention: The evaluation experience took place during a ten-month period between December 2012 and October 2013. This paper provides a summary of the context and a justification for the approach.
Research Design: Not applicable.
Data Collection and Analysis: In additional to conventional data collection instruments, we applied participatory inquiry tools from Social Analysis Systems (www.sas2.net) as a means of engaging stakeholders in real-time data collection and analysis.
Findings: Examples are provided to demonstrate how some developmental evaluation findings informed organizational strategic and operational decisions during the life of the evaluation.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
Funding data
References
Bryson, J.M.; Patton, M.Q., & Bowman, R.A. (2011). Working with evaluation stakeholders: A rationale, step-wise approach and toolkit. Evaluation and program planning 34: 1-2 https://doi.org/10.1016/j.evalprogplan.2010.07.001 DOI: https://doi.org/10.1016/j.evalprogplan.2010.07.001
Cook, T. (2006). Collaborative action research within developmental evaluation. Evaluation 12(4): 418-436 https://doi.org/10.1177/1356389006071293 DOI: https://doi.org/10.1177/1356389006071293
Gamble, J. (2008). A developmental evaluation primer. Montreal: The J.W. McConnell Family Foundation.
Gamble, J. (2006). Emerging learning about developmental evaluation. Hampton, NB, Canada: Imprint.
Horton, D.; Alexaki, A.; Bennett-Lartey, S.; Brice, K.N.; Campilan, D.; Carden, F.; Silva, J. de Souza; Duong, L.T.; Khadar, I.; Maestrey, Boza, A.; Kayes Muniruzzaman, I.; Perez, J.; Somarriba, Chang, M.; Vernooy, R. and Watts, J. (2003). Evaluating capacity development: Experiences from research and development organizations around the world. The Hague, Wageningen, Ottawa, ISNAR, CTA, IDRC.
Kania, J. & Kramer, M. (2013). Embracing emergence: How collective impact addresses complexity. Stanford Social Innovation Review. Jan 21 blog. http://www.ssireview.org/blog/entry/embracing_emergence_how_collective_impact_addresses_complexity (accessed 28 Aug. '13)
Langlois, M.; Blanchet-Cohen, N. & Beer, N. (2013). The art of the nudge: Five practices in developmental evaluation. The Canadian Journal of Program Evaluation 27(2): 39-59. https://doi.org/10.3138/cjpe.27.003 DOI: https://doi.org/10.3138/cjpe.27.003
Lennie, J. & Tacchi, J. (2013). Evaluating communication for development: A framework for social change. Abingdon, Oxon, UK & New York: Routledge. https://doi.org/10.4324/9780203078495 DOI: https://doi.org/10.4324/9780203078495
Leonard, D. (2013). Negotiating evaluation design in developmental evaluation: An emerging framework for shared decision-making. Panel presentation. Canadian Evaluation Society Conference Evaluation Across Boundaries, Toronto, June 9-12
Liket, K.; Rey-Garcia, M.; & Maas, K. (2014). Why aren't evaluations working and what to do about it: A framework for negotiating meaningful evaluation in nonprofits. American Journal of Evaluation 35(2): 171-188. https://doi.org/10.1177/1098214013517736 DOI: https://doi.org/10.1177/1098214013517736
Macaulay A. C.; Commanda, L.E.; Freeman, W.L.; Gibson, N.; McCabe M.L. & Robbins, C.M. (1999). Participatory research maximises community and lay involvement. BMJ 1999; 319:774 http://www.bmj.com/content/319/7212/774 https://doi.org/10.1136/bmj.319.7212.774 DOI: https://doi.org/10.1136/bmj.319.7212.774
McConnell Foundation. (2006). Sustaining social innovation: Developmental Evaluation. Montreal: The J.W. McConnell Family Foundation.
Mercer, J. (2007). The challenges of insider research in educational institutions: wielding a double edged sword and resolving delicate dilemmas. Oxford Review of Education, 33(1), 1-17. https://doi.org/10.1080/03054980601094651 DOI: https://doi.org/10.1080/03054980601094651
Parsons, H. M.(1974). What Happened at Hawthorne?: New evidence suggests the Hawthorne effect resulted from operant reinforcement contingencies. Science, 183(4128), 922-932. https://doi.org/10.1126/science.183.4128.922 DOI: https://doi.org/10.1126/science.183.4128.922
Patton, M.Q. (2012). Essentials of utilization-focused evaluation. Sage.
Patton, M.Q. 2011. Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York and London: The Guildford Press.
Patton, M.Q. (2008). Utilization-focused evaluation. Sage.
Patton, M.Q. (1994). Developmental evaluation. Evaluation practice 15(3): 311-319 https://doi.org/10.1177/109821409401500312 DOI: https://doi.org/10.1177/109821409401500312
Preskill, H.S. & Beer, T. (2012). Evaluating social innovation. Boston and Washington, DC: FSG and Center for Evaluation Innovation. https://doi.org/10.22163/fteval.2012.119 DOI: https://doi.org/10.22163/fteval.2012.119
Preskill, H.S. & Jones, N. (2009). A practical guide to engaging stakeholders in developing evaluation questions. Robert Wood Johnson Foundation Series. Princeton, NJ.
Ramírez, R. (2013). Exploring the "L" in evaluation: Putting the learning at the centre. Member of the panel session, the Canadian Evaluation Society Conference, Toronto 10-12 June.
Ramírez, R. & Brodhead, D.P. (2013). Utilization focused evaluation: A primer for evaluators. Penang: Southbound.
Saunders, M. (2012). The use and usability of evaluation outputs: A social practice approach. Evaluation 18(4): 421-436. https://doi.org/10.1177/1356389012459113 DOI: https://doi.org/10.1177/1356389012459113
Schwandt, T. (2014). On the mutually informing relationship between practice and theory in evaluation. American Journal of Evaluation 35(2): 231-236. https://doi.org/10.1177/1098214013503703 DOI: https://doi.org/10.1177/1098214013503703
Shephard, D. & Bailey, S. (2014). Combining utilization-focused evaluation and developmental evaluation: The case of the Aflateen Social and Financial Education Program for Youth. Case study presentation at the CDI Conference: Improving the use of M&E processes and findings. 20-21 March, Wageningen, the Netherlands. http://www.managingforimpact.org/sites/default/files/case/cdi_me_seminar_2014_background_paper_aflatoun.pdf
Stern, E.; Stame, N.; Mayne, J.; Forss, K.; Davies, R. and Befani, B. (2012). Broadening the range of designs and methods for impact evaluations. Report of a study commissioned by the Department of International Development, DFID Working Paper 38. London: DIFD. https://doi.org/10.22163/fteval.2012.100 DOI: https://doi.org/10.22163/fteval.2012.100
Westley, F.; Zimmerman, B. and Patton, M.Q. (2006). Getting to maybe: How the world is changed. Toronto: Random House.
Williams, A. (2014). Evaluation for strategic learning: Assessing readiness and results. Washington, DC: Center for Evaluation Innovation. http://www.evaluationinnovation.org/sites/default/files/Williams%20Strategic%20Learning.pdf