Learning How to Learn in Sustainability Transitions Projects: The Potential Contribution of Developmental Evaluation
Main Article Content
Abstract
Background: Community-based sustainability transitions projects are increasingly being considered for their potential as policy delivery vehicles for the UK government Climate Change Act commitments. At the same time, project funders seek reassurances that their investments are relevant in helping communities mitigate, and adapt to, the effects of climate change. Despite this increased pressure, recent research suggests that evaluations of such sustainability transitions projects have, on the one hand variable impacts, or impacts that are of an inadequate scale, duration, or type, or on the other, that project staff lack the capacity or resources to undertake monitoring and evaluation to the degree of rigour expected by policy makers and funders.
Purpose: This article reports on an extended case study of a fully-funded five year community-based sustainability transitions project in Leicestershire, England. In particular, it reviews the deployment of developmental evaluation (DE) methods in an attempt to capture the project team’s learning about doing community-based sustainability work.
Setting: A funded community-based sustainability transitions project in a south Leicestershire market town.
Intervention: Developmental evaluation methods were used to capture project-based learning as a resource for project innovation and adaptation.
Research Design: Phronetic case study.
Data Collection and Analysis: Participant-observation, action research, focus and special issue group facilitation.
Findings: Use of a developmental evaluation method identified key learning points for the project actors; these focused on how the project had adapted to the complexities of the operating environment through innovations in second-order learning or learning how to learn. The paper makes recommendations for the design and funding arrangements of community-based sustainability transitions initiatives and developmental evaluation is endorsed as a viable and promising adjunct to more traditional impact, economic, and process evaluation methodologies.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Abrahamse, W., Steg, L., Vlek, C. & Rothengatter, T. (2005). A review of intervention studies aimed at household energy conservation. Journal of Environmental Psychology, 25(3), 273-291. https://doi.org/10.1016/j.jenvp.2005.08.002. DOI: https://doi.org/10.1016/j.jenvp.2005.08.002
Aiken, G. T. (2015). (Local-) community for global challenges: carbon conversations, transition towns and governmental elisions. Local Environment, 20(17), 764-781. https://doi.org/10.1080/13549839.2013.870142. DOI: https://doi.org/10.1080/13549839.2013.870142
Bamberg, S., & Möser, G. (2007) Twenty years after Hines, Hungerford, and Tomera: A new meta-analysis of psycho-social determinants of pro-environmental behavior. Journal of Environmental Psychology, 27(1), 14-25. https://doi.org/10.1016/j.jenvp.2006.12.002. DOI: https://doi.org/10.1016/j.jenvp.2006.12.002
Beers, P. J., Hermans, F., Veldkamp, T., & Hinssen, J. (2014), Social learning inside and outside transition projects: Playing free jazz for a heavy metal audience. NJAS - Wageningen Journal of Life Sciences, 69, 5-13. https://doi.org/10.1016/j.njas.2013.10.001. DOI: https://doi.org/10.1016/j.njas.2013.10.001
Benford, R. D., & Snow, D. (2000). Framing processes and social movements: An overview and assessment. Annual Review Sociologic, 26, 611-639. https://doi.org/10.1146/annurev.soc.26.1.611. DOI: https://doi.org/10.1146/annurev.soc.26.1.611
Blamey, A., & Mackenzie, M. (2007) Theories of change and realistic evaluation: Peas in a pod or apples and oranges? Evaluation, 13(4), 439–-55. https://doi.org/10.1177/1356389007082129. DOI: https://doi.org/10.1177/1356389007082129
Bolderdijk, J. W., Steg, L., Geller, E. S., Lehman, P. K., & Postmes, T. (2013). Comparing the effectiveness of monetary versus moral motives in environmental campaigning. Nature Climate Change, 3(4), 413-416. https://doi.org/10.1038/nclimate1767. DOI: https://doi.org/10.1038/nclimate1767
Bradley, Q. (2014). Bringing democracy back home: Community localism and the domestication of political space. Environment & Planning D: Society & Space, 32(4), 642-657. https://doi.org/10.1068/d17312. DOI: https://doi.org/10.1068/d17312
Burns, D. (2010). Systematic action research: A strategy for whole system change. Bristol, UK: The Policy Press.
Burns, D., & Worsley, S. (2015). Navigating complexity in international development: Facilitating sustainable change at scale. Rugby: Practical Action Publishing Ltd. https://doi.org/10.3362/9781780448510.000 DOI: https://doi.org/10.3362/9781780448510.000
Checkland, P., & Holwell, S. (2007). Action research: Its nature and validity. Information Systems Action Research: An Applied View of Emerging Concepts and Methods, 11(1), 3-17. https://doi.org/10.1023/A:1022908820784. DOI: https://doi.org/10.1007/978-0-387-36060-7_1
Connell, J. P., & Kubisch, A. C. (1998). Applying a theory of change approach to the evaluation of comprehensive community initiatives: Progress, prospects, and problems. In K. Fulbright-Anderson, A. C. Kubisch, & J. P. Connell, J. P. (Eds.), New approaches to evaluating community initiatives. Volume 2: Theory, measurement, and analysis (pp. 15-44). Washington, DC: Aspen Institute.
Cress, D. M., & Snow, D. (2000). The outcomes of homeless mobilization: The influence of organization, disruption, political mediation and framing. American Journal of Sociology, 105(4), 1063-1104. https://doi.org/10.1086/210399. DOI: https://doi.org/10.1086/210399
Dahl, A. L. (2012). Achievements and gaps in indicators for sustainability. Ecological Indicators, 17, 14-19. https://doi.org/10.1016/j.ecolind.2011.04.032. DOI: https://doi.org/10.1016/j.ecolind.2011.04.032
DECC. (2014) Community energy strategy: full report.
Dunkley, R. A., & Franklin, A. (2017). Failing better: The stochastic art of evaluating community-led environmental action programs. Evaluation and Program Planning, 60, 112-122. https://doi.org/10.1016/j.evalprogplan.2016.11.005. DOI: https://doi.org/10.1016/j.evalprogplan.2016.11.005
Gooding, K. (2016). What do we mean by evidence-based advocacy? Ideas from NGOs in Malawi'. In R. Hayman, S. King, T. Kontinen, & L. Narayanaswamy (Eds.), Negotiating knowledge: Evidence and experience in development NGOs (pp. 15-28). Rugby: Practical Action Publishing Ltd. https://doi.org/10.3362/9781780449258.002 DOI: https://doi.org/10.3362/9781780449258.002
Gupta, R., Eyre, N., Darby, S., Lucas, K., Barnfield, L., Hamilton, J., Mayne, R., Gregg, M., Fratter, C., & Irving, B. (2015). Evaluating the impacts, effectiveness and success of low carbon communities on localised energy behaviours (EVALOC) Final Report. Oxford Brookes University and University of Oxford, Oxford. Available at: http://media.wix.com/ugd/caf2de_c6393816adeb44e8862c11f1dabd276b.pdf.
H. M. Government. (2011). Localism Act 2011 (c.20). UK.
Hamilton, J. (2013). Community monitoring and evaluation methods survey: Report and analysis. Evaluating low carbon communities: Working papers and reports. Available at: http://media.wix.com/ugd/caf2de_fd61a89150c88692722b9b0ea329ec98.pdf (Accessed: 12 December 2014).
Hargreaves, T., Hielscher, S., Seyfang, G., & Smith, A. (2013). Grassroots innovations in community energy: The role of intermediaries in niche development. Global Environmental Change, 23(5), 868-880. https://doi.org/10.1016/j.gloenvcha.2013.02.008. DOI: https://doi.org/10.1016/j.gloenvcha.2013.02.008
Hobson, K., Hamilton, J., & Mayne, R. (2014). Monitoring and evaluation in UK low-carbon community groups : benefits, barriers and the politics of the local. Local Environment : The International Journal of Justice and Sustainability, 21(1), 37-41. https://doi.org/10.1080/13549839.2014.928814. DOI: https://doi.org/10.1080/13549839.2014.928814
Hobson, K., Mayne, R., & Hamilton, J. (2016). Monitoring and evaluating eco-localisation: Lessons from UK low carbon community groups. Environment and Planning A, 48(7): 1393-1410. https://doi.org/10.1177/0308518X16640531. DOI: https://doi.org/10.1177/0308518X16640531
Hoffman, S. M., & High-Pippert, A. (2010). From private lives to collective action: Recruitment and participation incentives for a community energy program. Energy Policy, 38(12), 7567-7574. https://doi.org/10.1016/j.enpol.2009.06.054. DOI: https://doi.org/10.1016/j.enpol.2009.06.054
Huang, R. (2014). RQDA: R-based qualitative data analysis. Available at http://rqda.r-forge.r-project.org/.
Ison, R., Blackmore, C., Collins, K., & Furniss, P. (2007). Systemic environmental decision making: designing learning systems. Kybernetes, 36(9/10), 1340-1361. https://doi.org/10.1108/03684920710827346. DOI: https://doi.org/10.1108/03684920710827346
Kern, F., & Smith, A. (2008). Restructuring energy systems for sustainability? Energy transition policy in the Netherlands. Energy Policy, 36(11), 4093-4103. https://doi.org/10.1016/j.enpol.2008.06.018. DOI: https://doi.org/10.1016/j.enpol.2008.06.018
Letcher, R., Roberts, S., & Redgrove, Z. (2007). Mobilising individual behavioural change through community initiatives: Lessons for tackling climate change, the energy review study. Available at https://www.cse.org.uk/downloads/reports-and-publications/community-energy/behaviour-change/mobilising-individual-behavioural-change-through-community-summary.pdf.
McDonald, H. (2016). Developmental evaluation : A tool to support innovation. Evaluation Matters—He Take Tō Te Aromatawai, 2, 79-97. https://doi.org/10.18296/em.0012. DOI: https://doi.org/10.18296/em.0012
Milligan, S., Coulton, C., York, P., & Register, R. (1998). Implementing a theory of change evaluation in the Cleveland community-building initiative: A case study. In K. Fulbright-Anderson, A. C. Kubisch, & J. P. Connell (Eds.), New approaches to evaluating community initiatives. Volume 2: Theory, measurement, and analysis (pp. 45-85). Washington, DC: Aspen Institute.
Mitchell, A. S. (2019). Second-order learning in developmental evaluation: New methods for complex conditions. Cham, Switzerland: Palgrave Macmillan. https://doi.org/10.1007/978-3-319-99371-3 DOI: https://doi.org/10.1007/978-3-319-99371-3
Mitchell, A. S., Lemon, M., & Fletcher, G. (2019). What's being tested and what's being learnt? A contribution to lessons learned evaluation methods for community-based sustainability initiatives. Central European Review of Economics and Management, 3(3), 129-168. https://doi.org/10.29015/cerem.596 DOI: https://doi.org/10.29015/cerem.596
Mowles, C. (2014). Complex, but not quite complex enough: The turn to the complexity sciences in evaluation scholarship. Evaluation, 20(2), 60-75. https://doi.org/10.1177/1356389014527885. DOI: https://doi.org/10.1177/1356389014527885
Patton, M. Q. (2001). Evaluation, knowledge management, best practices, and high quality lessons learned. American Journal of Evaluation, 22, 329-336. https://doi.org/10.1177/109821400102200307. DOI: https://doi.org/10.1016/S1098-2140(01)00147-3
Patton, M. Q. (2011). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York, NY: Guildford.
Petersen, J. E., Shunturov, V., Janda, K., Platt, G., & Weinberger, K. (2007). Dormitory residents reduce electricity consumption when exposed to real-time visual feedback and incentives. International Journal of Sustainability in Higher Education, 8(1), 16-33. https://doi.org/10.1108/14676370710717562. DOI: https://doi.org/10.1108/14676370710717562
Seyfang, G. (2010). Community action for sustainable housing: Building a low-carbon future. Energy Policy, 38, 7624-7633. https://doi.org/10.1016/j.enpol.2009.10.027. DOI: https://doi.org/10.1016/j.enpol.2009.10.027
Seyfang, G., Hielscher, S., Hargreaves, T., Martiskainen, M., & Smith, A. (2014). Environmental innovation and societal transitions: A grassroots sustainable energy niche? Reflections on community energy in the UK. Environmental Innovation and Societal Transitions. https://doi.org/10.1016/j.eist.2014.04.004. DOI: https://doi.org/10.1016/j.eist.2014.04.004
Steg, L., & Vlek, C. (2009). Encouraging pro-environmental behaviour: An integrative review and research agenda. Journal of Environmental Psychology, 29(3), 309-317. https://doi.org/10.1016/j.jenvp.2008.10.004. DOI: https://doi.org/10.1016/j.jenvp.2008.10.004
Stephenson, J., Barton, B., Carrington, G., Gnoth, D., Lawson, R., & Thorsnes, P. (2010) Energy cultures: A framework for understanding energy behaviours. Energy Policy, 38, 6120-6129. https://doi.org/10.1016/j.enpol.2010.05.069. DOI: https://doi.org/10.1016/j.enpol.2010.05.069
TESS. (2016). Success factors of community-based sustainability initiatives—Results from TESS. Available at: http://www.tess-transition.eu/wp-content/uploads/2016/10/Tess_draft_Policy-Brief_success_1-1.pdf.
Turcu, C. (2013). Re-thinking sustainability indicators: Local perspectives of urban sustainability. Journal of Environmental Planning and Management, 56(5), 695-719. https://doi.org/10.1080/09640568.2012.698984. DOI: https://doi.org/10.1080/09640568.2012.698984
UK Parliament. (2008). Climate Change Act. https://doi.org/10.1136/bmj.39469.569815.47. DOI: https://doi.org/10.1136/bmj.39469.569815.47
Weiss, C. H. (1995). Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. In J. Connell, A. Kubisch, L. Schorr, & C. Weiss (Eds.), New Approaches to Evaluating Community Initiatives (pp. 65-92). Washington, DC: Aspen Institute.
Weiss, C. H. (1997). How can theory-based evaluation make greater headway? Evaluation Review, 21(4), 501-524. https://doi.org/10.1177/0193841X9702100405. DOI: https://doi.org/10.1177/0193841X9702100405