A Complexity-Based Plan for Evaluating Transformation
Article Sidebar
Main Article Content
Abstract
This article presents a case for more rigorous application of complexity science in our efforts to evaluate activity that seeks to bring about transformative change. It builds on the work that is already going on in the evaluation community. Three constructs from complexity science are employed – sensitive dependence, emergence, and social attractors. The paper argues that if–then logic is recommended for small-scale change within transformation efforts, but that to evaluate transformation writ large, data from if–then evaluation must be embedded in, and interpreted in terms of, complex behavior. Methodologies for evaluating within this framework are presented. The argument is linked to a definition of transformation that is multidimensional, non-linear, and measurable. The paper is built around a generic model of transformational change and shows how that model can be customized for specific transformation scenarios. It also shows how evaluation with respect to complexity can be accomplished with methodologies that are well known and well-practiced in the evaluation community.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Acemoglu, D., & Robinson, J. A. (2012). Why nations fail. Currency / Random House. https://www.penguinrandomhouse.com/books/205014/why-nations-fail-by-daron-acemoglu-and-james-a-robinson/
Anand, A. (2020, second quarter). Integrating big geodata and technology in evaluation: What do we need to know? Evaluation Matters Magazine. https://idev.afdb.org/en/document/preparing-evaluation-future-big-data-modern-technologies-and-shifts-global-development
Anguko, A. (2020). Application of SurveyCTO mobile data collection technology in household surveys: The case of an impact evaluation of the Community Based Integrated Water Resource Management Project in Niger Evaluation Matters Magazine. https://idev.afdb.org/en/document/preparing-evaluation-future-big-data-modern-technologies-and-shifts-global-development DOI: https://doi.org/10.21201/2019.4641
Bamberger, M., & York, P. (2020, second quarter). Transforming evaluation in the 4th industrial revolution: Exciting opportunities and new challenges. Evaluation Matters Magazine. https://idev.afdb.org/en/document/preparing-evaluation-future-big-data-modern-technologies-and-shifts-global-development
Bar-Yam, Y. (2021). Concept map. New England Complex Systems Institute. https://necsi.edu/concept-map
Barabási, A.-L., & Albert, R. (1999). Emergence of scaling in random networks. Science, 286(5439), 509. https://doi.org/10.1126/science.286.5439.509 DOI: https://doi.org/10.1126/science.286.5439.509
Barbrook-Johnson, P., Proctor, A., Giorgi, S., & Phillipson, J. (2020). How do policy evaluators understand complexity? Evaluation, 26(3), 315–332. https://doi.org/10.1177/1356389020930053 DOI: https://doi.org/10.1177/1356389020930053
Batra, G., Garcia, J., & Temnenko, K. (2022). Transformational change for achieving scale: Lessons for a greener recovery. In J. I. Uitto & G. Battra (Eds.), Transformational change for people and the planet: Evaluating environment and development (pp. 27–38). Springer. DOI: https://doi.org/10.1007/978-3-030-78853-7_3
Bicket, M., Christie, I., Gilbert, N., Hills, D., Penn, A., & Wilkinson, H. (2020). Supplementary guide: Handling complexity in policy evaluation (HM Treasury Ed.). CECAN—the Centre for Evaluating Complexity Across the Nexus. https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/879437/Magenta_Book_supplementary_guide._Handling_Complexity_in_policy_evaluation.pdf
Box, G. E. P. (1979). Robustness in the strategy of scientific model building. In R. L. Launer & G. N. Wilkinson (Eds.), Robustness in Statistics (pp. 201–236). Academic Press. DOI: https://doi.org/10.1016/B978-0-12-438150-6.50018-2
Byrne, D., & Uprichard, E. (2012). Useful complex causality. In H. Kinkard (Ed.), Oxford handbook of philosophy of social science (pp. 109–129). Oxford University Press. https://doi.org/https://doi.org/10.1093/oxfordhb/9780195392753.013.0006 DOI: https://doi.org/10.1093/oxfordhb/9780195392753.013.0006
Castellani, B., & Gerrits, L. (2021). 2021 map of the complexity sciences. Art & Science Factory. https://www.art-sciencefactory.com/complexity-map_feb09.html
Comfort, L. K. (1994). Self-Organization in complex systems. Journal of Public Administration Research and Theory, 4(3), 393–410. http://www.jstor.org/stable/1181895
Egré, P., & Rott, H. (2021). The logic of conditionals. In E. N. Zalta (Ed.), The Stanford encyclopedia of philosophy. Stanford University.
Feinstein, O. (2019). Dynamic evaluation for transformational change. In R. D. van den Berg, C. Magro, & S. S. Mulder (Eds.), Evaluation for transformational change: Opportunities and challenges for the Sustainable Development Goals (pp. 17–32). International Development Evaluation Association (IDEAS). https://ideas-global.org/wp-content/uploads/2019/11/2019-11-05-Final_IDEAS_EvaluationForTransformationalChange.pdf
Firmansyah, R., Putri, D., Wicaksono, M., Putri, S., Widianto, A., & Palil, M. (2021). Educational transformation: An evaluation of online learning due To COVID-19. International Journal of Emerging Technologies in Learning (iJET), 16(7), 61–76. DOI: https://doi.org/10.3991/ijet.v16i07.21201
Fisher, S., & Roehrer, C. (2020). Understanding progress towards transformation, in time. EIT Climate-KIC.
Frigg, R., & Hartmann, S. (2018). Models in Science. In E. N. Zalta (Ed.), The Stanford encyclopedia of philosophy. Stanford University. https://plato.stanford.edu/archives/sum2018/entries/models-science/
Gates, E. F., Walton, M., Vidueira, P., & McNall, M. (2021). Introducing systems- and complexity-informed evaluation. New Directions for Evaluation, 170, 13–25. https://doi.org/https://doi.org/10.1002/ev.20466 DOI: https://doi.org/10.1002/ev.20466
Glaeser, E. (2012). Triumph of the city: How our greatest invention makes us richer, smarter, greener, healthier, and happier. Penguin Group. DOI: https://doi.org/10.17323/1726-3247-2013-4-75-94
Gordon, T. J., & Glenn, J. C. (2009). Environmental scanning. In T. J. Gordon & J. C. Glenn (Eds.), Futures research methodology — version 3.0. The Millennium Project. https://www.researchgate.net/profile/Jerome-Glenn/publication/286279000_Environmental_Scanning/links/5667954a08aea62726ee9541/Environmental-Scanning.pdf
Greenhalgh, T., Humphrey, C., Hughes, J., Macfarlane, F., Butler, C., & Pawson, R. (2009). How do you modernize a health service? A realist evaluation of whole-scale transformation in London. The Milbank Quarterly, 87(2), 391–416. https://doi.org/https://doi.org/10.1111/j.1468-0009.2009.00562.x DOI: https://doi.org/10.1111/j.1468-0009.2009.00562.x
Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: Systematic review and recommendations. The Milbank Quarterly, 82(4), 581–629. https://doi.org/10.1111/j.0887-378X.2004.00325.x DOI: https://doi.org/10.1111/j.0887-378X.2004.00325.x
Ibrahim, M., El-Zaart, A., & Adams, C. (2017, September 12–14). Theory of change for the transformation towards smart sustainable cities [Paper presentation]. Sensors Networks Smart and Emerging Technologies (SENSET). Beiriut, Lebanon DOI: https://doi.org/10.1109/SENSET.2017.8125067
Initiative for Climate Action Transparency (ICAT). (2020). Transformational change methodology: Assessing the transformational impacts of policies and actions. https://www.iges.or.jp/en/pub/transformational-change-methodology/en
International Evaluation Group. (2016). Supporting transformational change for poverty reduction and shared prosperity. World Bank. https://ieg.worldbankgroup.org/evaluations/supporting-transformational-change-poverty-reduction-and-shared-prosperity
Jeong, H., Néda, Z., & Barabási, A. L. (2003). Measuring preferential attachment in evolving networks. Europhysics Letters (EPL), 61(4), 567–572. https://doi.org/10.1209/epl/i2003-00166-9 DOI: https://doi.org/10.1209/epl/i2003-00166-9
Lamberson, P. J., & Page, S. E. (2012). Tipping points (Paper #12-02-002). Santa Fe Institute. https://www.santafe.edu/research/results/working-papers/tipping-points
Leo, B., Pattni, S., Winn, C., Lewis, Q., Paton, C., & Fraym, M. P. (2020, second quarter). Using machine learning for climate related impact evaluations. Evaluation Matters Magazine. https://idev.afdb.org/en/document/preparing-evaluation-future-big-data-modern-technologies-and-shifts-global-development
Matz, R. L., Fata-Hartley, C. L., Posey, L. A., Laverty, J. T., Underwood, S. M., Carmel, J. H., Herrington, D. G., Stowe, R. L., Caballero, M. D., Ebert-May, D., & Cooper, M. M. (2018). Evaluating the extent of a large-scale transformation in gateway science courses. Science Advances, 4(10), eaau0554. https://doi.org/doi:10.1126/sciadv.aau0554 DOI: https://doi.org/10.1126/sciadv.aau0554
Morell, J. A. (2021). A complexity-based metatheory of action for transformation to a green energy future. In R. D. Van den Berg, C. Maro, & M.-H. Adrien (Eds.), Transformational evaluation for the global crises of our times (pp. 345–370). IDEAS. https://ideas-global.org/ideas-book-transformational-evaluation-for-the-global-crises-of-our-times/
Ofir, Z., & Rugg, D. (2021). International developments in evaluation: Transforming evaluation [special section]. American Journal of Evaluation, 42(1), 47–138. DOI: https://doi.org/10.1177/1098214020979070
Okpe, B. (2020, second quarter). Integrating big data analytics and artificial intelligence into monitoring and evaluation in a fast-changing development landscape. Evaluation Matters Magazine. https://idev.afdb.org/en/document/preparing-evaluation-future-big-data-modern-technologies-and-shifts-global-development
Parunak, H. V. D. (2023). Analyzing the Behavioral Attractors of a Social Simulation. In P. A. Research, Model Mechanisms and Behavioral Attractors. Social Simulation Conference (SSC2022), Milan. https://www.abcresearch.org/abc/papers/SSC2022Attractors.pdf.
Patton, M. Q. (2019). Transformation to global sustainability: Implications for evaluation and evaluators. New Directions for Evaluation, 162, 103–117. https://doi.org/10.1002/ev.20362 DOI: https://doi.org/10.1002/ev.20362
Patton, M. Q. (2020). Blue marble evaluation: Premises and principles. Guilford Press. DOI: https://doi.org/10.31244/zfe.2020.02.09
Pearl, J. (2009). Causality: Models, reasoning and inference (2nd ed.). Cambridge University Press. DOI: https://doi.org/10.1017/CBO9780511803161
Phelan, S. E. (2001). What is complexity science, really? Emergence, 3(1), 120–136. https://doi.org/https://doi.org/10.1207/S15327000EM0301_08 DOI: https://doi.org/10.1207/S15327000EM0301_08
Picciotto, R. (2021). Toward a complexity framework for transformative evaluation. Journal.of MultiDisciplinary Evaluation, 16(35), 54–76. DOI: https://doi.org/10.56645/jmde.v16i35.643
Rogers, E. (2003). Diffusion of innovations (5th ed.). Free Press.
Rogers, K. (2012). Scientific modeling. Encyclopædia Britannica. https://www.britannica.com/science/scientific-modeling
Santa Fe Institute. (n.d.-a). Attractor. Complexity Explorer. https://www.complexityexplorer.org/explore/glossary/6-attractor
Santa Fe Institute. (n.d.-b). Emergence. Complexity Explorer. https://www.complexityexplorer.org/explore/glossary/414-emergence
Santa Fe Institute. (n.d.-c). Sensitive dependence. Complexity Explorer. https://www.complexityexplorer.org/explore/glossary/44-sensitive-dependence-on-initial-conditions
SDG Transformation Forum. (2020). Our theory of transformational change. SDG Transformation Forum.
Statistical model. (2023, June 30). In Wikipedia. https://en.wikipedia.org/w/index.php?title=Statistical_model&oldid=1161602898
Systems Innovation. (2020a). Attractor. YouTube. https://www.youtube.com/watch?v=C9Xl6BNu-kM
Systems Innovation. (2020b). Social attractors & chaos [Video]. YouTube. https://www.youtube.com/watch?v=C9Xl6BNu-kM
Theraulaz, G., & Bonabeau, E. (1999). A Brief history of stigmergy. Artificial Life, 5, 97–116. DOI: https://doi.org/10.1162/106454699568700
U.S. Department of Energy. (2012). DOE handbook: Accident and operational safety analysis: Vol. 1. Accident analysis techniques (DOE‐HDBK‐1208‐2012). https://www.standards.doe.gov/standards-documents/1200/1208-bhdbk-2012-v1/@@images/file
Van den Berg, R. D., Maro, C., & Adrien, M.-H. (Eds.). (2021). Transformational evaluation for the global crises of our times. International Development Evaluation Association (IDEA). https://ideas-global.org/wp-content/uploads/2021/07/2021-IDEAS-book-Transformational-Evaluation.pdf.
Wadeson, A., Monzani, B., & Aston, T. (2020 ). Process tracing as a practical evaluation method: Comparative learning from six evaluations. https://mande.co.uk/wp-content/uploads/2020/03/Process-Tracing-as-a-Practical-Evaluation-Method_23March-Final.pdf
West, G. (2017). Scale: The universal laws of growth, innovation, sustainability, and the pace of life in organisms, cities, economies, and companies. Penguin.
Zazueta, A. (2017). Principles for the development of integrated transformational projects in climate change and chemicals & waste. Global Environment Facility. http://www.stapgef.org/sites/default/files/publications/Final_Principles%20for%20%20Development%20of%20Integrated%20CCM%20and%20ChW%20Projects_October%202017.pdf
Zazueta, A. (2019). Using theories of change to steer transformational development [Paper presentation]. IDEAS Global Assembly / 3rd International Conference on Evaluating Environment and Development, Prague, Czech Republic.
Zellner, M. L., Lyons, L. B., Hoch, C. J., Weizeorick, J., Kunda, C., & Milz, D. C. (2012). Modeling, learning, and planning together: An application of participatory agent-based modeling to environmental planning. URISA Journal, 24(1), 77–92. https://www.academia.edu/download/47465368/Zellner_2012.pdf