Complexity-Aware Evaluation for Learning: A Case Study of a Developmental Approach
Article Sidebar
Main Article Content
Abstract
Background: Ongoing learning is vital for steering programs in complex social-ecological systems, because it enables implementers to change course when needed, and ideally contributes to system wide insights as well. While there are many calls and guidelines for complexity-aware and learning-centred monitoring and evaluation (M&E), there are fewer case examples of attempts to implement it in practice.
Purpose: This paper reflects on a multi-year case of an attempt to design and implement a complexity-aware, developmental and learning-centred M&E framework in a complex socio-ecological landscape, highlighting the challenges, adaptations and evidence of learning.
Setting: Olifants River Basin, north-eastern South Africa and southern Mozambique.
Intervention: Resilience in the Limpopo (Olifants Basin) Program (RESILIM-O).
Research Design: Participatory action-reflection research with data collection through document analysis, interviews and focus groups
Findings: Learning among program implementers was facilitated by working in new ways with standard M&E elements including indicator-based targets and theories of change, and through opportunities for shared reflection, including novel reporting templates, reflection days, and collaborative case studies. Challenges notwithstanding, the participatory, developmental M&E approach built understanding of and competence in M&E in the organisation. The “hybrid” framework that evolved successfully combined the need for accountability with the desire for organisational learning; and associated features were subsequently adopted elsewhere.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
Funding data
References
AWARD (2020). RESILIM-Olifants: Resilience in the Limpopo Basin program final report. AWARD, Hoedspruit. https://award.org.za/wp/wp-content/uploads/2020/12/AWARD-RESILIM-Olifants-FINAL-REPORT-PUBLIC-Oct-2020-web.pdf
Bakewell, O. (2003). Sharpening the development process: A practical guide to monitoring and evaluation (Praxis Guide No. 1). INTRAC.
Bellamy, J. A., Walker, D. H., McDonald, G. T., & Syme, G. J. (2001). A systems approach to the evaluation of natural resource management initiatives. Journal of Environmental Management, 63, 407–423. https://doi.org/10.1006/jema.2001.0493
Boreham, N., & Morgan, C. (2004) A sociocultural analysis of organisational learning. Oxford Review of Education, 30(3), 307–325. https://doi.org/10.1080/0305498042000260467
Bornstein, L. (2006). Systems of accountability, webs of deceit? Monitoring and evaluation in South African NGOs. Development, 49(2), 52–61. https://doi.org/10.1057/palgrave.development.1100261
Chapman, S., Sullivan, C., Palm, C., Huynh, U., Diru, W., & Masira, J. (2016). Monitoring and evaluation to support adaptive co-management: Lessons learned from the Millennium Villages Project. Journal of Environmental Management, 183, 142–151. http://dx.doi.org/10.1016/j.jenvman.2016.08.014
Chikunda, C. (2017). Mediating sensemaking: Enriching social learning for resilience in natural resources management learning. Proceedings of the 9th Action Learning Action Research and 13th Participatory Action Research World Congress 2015. Action Learning Action Research Association (ALARA). https://www.alarassociation.org/publications/other-alara-publications/world-congress-proceedings-2015
Chirau, T. J., & Blaser-Mapitsa, C. (2020). How performance management regulations shape evaluation practice in South African municipalities. Evaluation and Program Planning, 82, Article 101831. https://doi.org/10.1016/j.evalprogplan.2020.101831
Cundill, G., & Fabricius, C. (2009). Monitoring in adaptive co-management: Toward a learning based approach. Journal of Environmental Management, 90(11), 3205–3211. https://doi.org/10.1016/j.jenvman.2009.05.012
Douthwaite, B., & Hoffecker, E. (2017). Towards a complexity-aware theory of change for participatory research programs working within agricultural innovation systems. Agricultural Systems, 155: 88–102. https://doi.org/10.1016/j.agsy.2017.04.002
Engeström, Y. (2016). Studies in expansive learning: Learning what is not yet there. Cambridge University Press. http://dx.doi.org/10.1017/CBO9781316225363
Engeström, Y., & Sannino, A. (2012). Whatever happened to process theories of learning? Learning, Culture and Social Interaction, 1(1), 45–56. https://doi.org/10.1016/j.lcsi.2012.03.002
Funnell, S. C., & Rogers, P. J. (2011). Purposeful program theory: Effective use of theories of change and logic models. John Wiley & Sons.
Hertz, T., Brattander, E., & Rose, L. (2021). Complexity-aware monitoring and evaluation. Journal of MultiDisciplinary Evaluation, 17(41), 35–50. https://journals.sfu.ca/jmde/index.php/jmde_1/article/view/679
Ison, R. L., Collins, K. B., & Iaquinto, B. L. (2021). Designing an inquiry-based learning system: Innovating in research praxis to transform science–policy–practice relations for sustainable development. Systems Research and Behavioural Science, 38(5), 610–624. https://doi.org/10.1002/sres.2811
Kachur, D., Soal, S., & van Blerk, R. (2016). Stretching between learning and accountability: Experiences of South African non-governmental organisations. African Evaluation Journal, 4(1), Article 71. http://dx.doi.org/10.4102/aej.v4i1.71
Khan, M. S., Rothman-Ostrow, P., Spencer, J., Hasan, N., Sabirovic, M., Rahman-Shepherd, A., Shaikh, N., Heymann, D. L., & Dar, O. (2018). The growth and strategic functioning of One Health networks: A systematic analysis. The Lancet Planetary Health, 2, e264–273. https://doi.org/10.1016/S2542-5196(18)30084-6
Lotz-Sisitka, H., Ali, M. B., Mphepo, G., Chaves, M., Macintyre, T., Pesanayi, T., Wals, A., Mukute, M., Kronlid, D., Tran, D. T., Joon, D., & McGarry, D. (2016). Co-designing research on transgressive learning in times of climate change. Current Opinion in Environmental Sustainability, 20, 50–55. http://dx.doi.org/10.1016/j.cosust.2016.04.004
Mushwana Mudau, V. (2020). A study on reporting and learning in three natural resource management programmes in South Africa [Unpublished master’s thesis]. Rhodes University.
Mueller-Hirth, N. (2012). If you don’t count, you don’t count: Monitoring and evaluation in South African NGOs. Development and Change, 43(3), 649–670. https://doi.org/10.1111/j.1467-7660.2012.01776.x
Natsios, A. (2010, July 1). The clash of the counter-bureaucracy and development. Center for Global Development, July 2010. https://www.cgdev.org/publication/clash-counter-bureaucracy-and-development
Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). SAGE.
Patton, M. Q. (2010). Developmental evaluation: Applying complexity concepts to enhance innovation and use. Guilford Press.
Patton, M. Q. (2021). Emergent developmental evaluation developments. Journal of MultiDisciplinary Evaluation, 17(41), 23–34. http://dx.doi.org/10.56645/jmde.v17i41.699
Pawson, R., & Tilley, N. (1997). Realistic evaluation. SAGE.
Pollard, S., du Toit, D., & Biggs, H. (2011). River management under transformation: The emergence of strategic adaptive management of river systems in the Kruger National Park. Koedoe, 53(2), 1–14. http://dx.doi.org/10.4102/koedoe.v53i2.1011
Pollard, S., Biggs, H., & du Toit, D. (2014). A systemic framework for context-based decision making in natural resource management: Reflections on an integrative assessment of water and livelihood security outcomes following policy reform in South Africa. Ecology and Society, 19(2), Article 63. http://dx.doi.org/10.5751/ES-06312-190263
Preiser, R., Biggs, R., de Vos, A., & Folke, C. (2018). Social-ecological systems as complex adaptive systems: Organizing principles for advancing research methods and approaches. Ecology and Society, 23(4), Article 46. https://doi.org/10.5751/ES-10558-230446
Pringle, P. (2011). AdaptME toolkit: Adaptation monitoring and evaluation. United Kingdom Climate Impacts Programme (UKCIP).
Reed, M. S., Evely, A. C., Cundill, G., Fazey, I., Glass, J., Laing, A., Newig, J., Parrish, B., Prell, C., Raymond, C., & Stringer, L. C. (2010). What is social learning? Ecology and Society, 15(4). http://dx.doi.org/10.5751/ES-03564-1504r01
Rosenberg, E., Kotschy, K., Burt, J., Mudau, V., du Toit, D., & Pollard, S. (2017). Monitoring, evaluation, reporting and learning for the USAID RESILIM-O program: MERL framework. AWARD. https://award.org.za/wp/wp-content/uploads/2021/04/AWARD-TECH-REPORT-18-RESILIM-O-Monitoring-Evaluation-Reporting-Learning-MERL-Framework-2017.pdf
Rosenberg, E, & Kotschy, K. (2020). Monitoring and evaluation in a changing world: A Southern African perspective on the skills needed for a new approach. African Evaluation Journal, 8(1), Article 472. https://doi.org/10.4102/aej.v8i1.472
Roux, D. J., Nel, J. L., Cundill, G., O’Farrell, P., & Fabricius, C. (2017). Transdisciplinary research for systemic change: Who to learn with, what to learn about and how to learn. Sustainability Science, 12(5), 711–726. https://doi.org/10.1007/s11625-017-0446-0
Schön, D. A. (1983). The reflective practitioner: How professionals think in action. Basic Books.
Schulz, K. (2010). Being wrong: Adventures in the margin of error. Portobello.
Scharmer, C. O. (2009). Theory U: Leading from future. Carl-Auer-Systeme Verlag.
Taylor, J., & Soal, S. (2011). Measurement in developmental practice: From the mundane to the transformational. In A. Fowler & C. Malunga (Eds.), NGO management (Chapter 26). Routledge. http://dx.doi.org/10.4324/9781849775427-34
Tsitsa Project, Kotschy, K., Cockburn, J., Conde-Aller, L., & Rosenberg, E. (2021). Participatory monitoring, evaluation, reflection and learning (PMERL): Building a participatory and sustainable system for evaluating impact (Tsitsa Project Practice and Policy Brief No. 5). Rhodes University. https://www.ru.ac.za/tsitsaproject/resourcesresearch/keyinformation/
USAID. (2021). Complexity-aware monitoring discussion note. United States Agency for International Development. https://usaidlearninglab.org/library/complexity-aware-monitoring-discussion-note-brief
Villanueva, P. S. (2011). Learning to ADAPT: Monitoring and evaluation approaches in climate change adaptation and disaster risk reduction—challenges, gaps and ways forward (Strengthening Climate Resilience Discussion Paper 9). Institute of Development Studies.
Walker, B., & Salt, D. (2006). Resilience thinking: Sustaining people and ecosystems in a changing world. Island Press.
Wals, A. E. (2007). Think Piece. Learning in a changing world and changing in a learning world: Reflexively fumbling towards sustainability. Southern African Journal of Environmental Education, 24, 35–45.
Weiss, C. H. (1997). Theory‐based evaluation: Past, present, and future. New Directions for Evaluation, 76, 41–55. https://doi.org/10.1002/ev.1086
Williams, B. K., & Brown, E. D. (2018). Double-loop learning in adaptive management: The need, the challenge, and the opportunity. Environmental Management, 62, 995–1006. https://doi.org/10.1007/s00267-018-1107-5
Wongtschowski, M., Oonk, L., & Mur, R. (2016). Monitoring and evaluation for accountability and learning (Working Paper 2016-3, Kit Sustainable Economic Development and Gender). https://www.kit.nl/wp-content/uploads/2019/10/WPS_3-2016-web.pdf
Woodhill, J. (2007). M&E as learning: Rethinking the dominant paradigm. In J. de Graaf, J. Camerson, S. Sombatpanit, C. Pieri, & J. Woodhill (Eds.), Monitoring and evaluation of social conservation and watershed development projects (pp. 83–107). Science Publishers.