Glocal Evaluation Competencies for Learning As We Go: Zooming in and zooming out to connect system-level solutions to local beneficiaries

Keiko Kuji-Shikatani; Charmagne Campbell-Patton; Wendy Rowe

doi:10.56645/jmde.v19i46.881

PDF

Published: Dec 11, 2023

DOI: https://doi.org/10.56645/jmde.v19i46.881

Keywords:

developmental evaluation, evaluator competency, Evaluation, evaluator competencies, interpersonal skills, evaluator education, evaluation practice, program evaluation, collaborative evaluation, Outcomes, utilization-focused evaluation, teaching evaluation, capacity building

Dr. Keiko Kuji-Shikatani, C.E.

Canadian Evaluation Society Fellow

Charmagne Campbell-Patton

Utilization-Focused Evaluation

Dr. Wendy Rowe, C.E.

Royal Roads University

Abstract

Identifying essential competencies for evaluators has received significant attention in recent years yet practical examples of how to apply competencies to real-time learning in complex environments are lacking. In particular, the experience of those at the local level - ultimate beneficiary individuals (UBIs) - can get lost when evaluations take a systems perspective. Experienced evaluators share how Learning as we go is used to describe utilization-focused developmental evaluation embedding evaluative thinking and building capacity in public sector programs, that support learning and adaptation to improve the lives of those most impacted by inequitable and unsustainable global systems.

Downloads

Download data is not yet available.

How to Cite

Kuji-Shikatani, K., Campbell-Patton, C., & Rowe, W. (2023). Glocal Evaluation Competencies for Learning As We Go: Zooming in and zooming out to connect system-level solutions to local beneficiaries. Journal of MultiDisciplinary Evaluation, 19(46), 31–44. https://doi.org/10.56645/jmde.v19i46.881

Issue

Vol. 19 No. 46 (2023): Special Issue: Evaluator Competencies

Section

International Evaluator Competence Perspectives

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Copyright and Permissions

Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org

Author Biographies

Dr. Keiko Kuji-Shikatani, C.E., Canadian Evaluation Society Fellow

Dr. Keiko Kuji-Shikatani, CESF, believes that evaluators-as-activist can contribute as stewards ensuring that generations to come will thrive in a caring, equitable and peaceful world. Devoted to supporting emerging evaluators and organizations through sharing her expertise on Learning as we go (DE) & evaluation capacity building in the public sector in Canada and internationally. Instrumental in the operationalization of the Credentialed Evaluator program which is documented in CJPE for VOPEs.

Charmagne Campbell-Patton, Utilization-Focused Evaluation

Charmagne is an independent consultant with fifteen years of program design, implementation and evaluation work across a range of fields. She specializes in qualitative research methods and utilization-focused developmental evaluation. She holds an MA in International Peace and Conflict Resolution from American University’s School of International Service and is co-author of the 5th Edition of Utilization-Focused Evaluation

Dr. Wendy Rowe, C.E., Royal Roads University

Wendy Rowe, CE, MA, PhD, is a scholar and organizational consultant with 25 years of experience working professionally in Canada, United States and international countries, using models of collaboration and empowerment evaluation to enhance organizational capacities, increase program efficiency and effectiveness, and create learning organizations. Dr. Rowe is currently a professor in the School of Leadership Studies at Royal Roads University, with over 150 reports and 30 academic publications.

References

ANZEA. (2011). Evaluator Competencies. Aotearoa New Zealand Evaluation Association. https://anzea.org.nz/assets/Key-ANZEA-Files/110801_anzea_evaluator_competencies_final.pdf

Aronsson, I., Hassnain, H., Lauren, K., & Somma, S. (2021). Evaluation in Contexts of Fragility, Conflict and Violence: Guidance from Global Evaluation Practitioners. International Development Evaluation Association (IDEAS).

Bolman, L. G., & Deal, T. E. (2017). Reframing organizations: Artistry, choice, and leadership. John Wiley & Sons. DOI: https://doi.org/10.1002/9781119281856

Blavathnik School of Government. (2022). Covid-19 Government Response Tracker. University of Oxford: https://www.bsg.ox.ac.uk/research/research-projects/covid-19-government-response-tracker

Blue Marble Evaluation. (2021, July 28). YouTube. https://www.youtube.com/watch?v=NbkWIMrXAmc

Buckley, J. et al., (2015). Defining and Teaching Evaluative Thinking: Insights From Research on Critical Thinking. American Journal of Evaluation. 36. 10.1177/1098214015581706. DOI: https://doi.org/10.1177/1098214015581706

Bukowski, H. (2019). Self-Knowledge. In V. Zeigler-Hill, & T. K. Shackelford, Encyclopedia of Personality and Individual Differences (pp. 1-5). Springer International Publishing. doi:10.1007/978-3-319-28099-8_2004-1 DOI: https://doi.org/10.1007/978-3-319-28099-8_2004-1

Canadian Evaluation Society. (2018). Competencies for Canadian Evaluation Practice. https://evaluationcanada.ca/txt/2_competencies_cdn_evaluation_practice_2018.pdf

Chi, Y., Kuji-Shikatani, K., & Love, A. (in progress). Meeting the Challenges of Educating Internal Evaluators. New Directions for Evaluation.

Clark, J., & Polesello, D. (2017). Emotional and cultural intelligence in diverse workplaces: getting out og the box. Industrial and Commercial Training, 49((7/8)), 337-349. https://doi.org/10.1108/ICT-06-2017-0040 DOI: https://doi.org/10.1108/ICT-06-2017-0040

Conto, A., Akseer, S., Dressen, T., Kamei, A., Mizunoya, S., & Rigole, A. (2020). Covid-19: Effects of School Closures on Foundational and Promising Parcites for Monitoring and Mitigating Learning Loss,. Innocenti Working Paper 2020-13, UNICEF Office of Research - Innocenti. https://www.unicef-irc.org/publications/pdf/COVID-19_Effects_of_School_Closures_on_Foundational_Skills_and_Promising_Practices_for_Monitoring_and_Mitigating_Learning_Loss.pdf

Daud, Y. (2020). Self-leadership and its application to today’s leader-A review of literature. Strategic Journal of Business Change Management 8(1), 1-11. DOI: https://doi.org/10.61426/sjbcm.v8i1.1887

Gerasimova, K. (n.d.). 2017. How to challenge your mental models and think differently. https://gothamculture.com/2017/03/23/challenge-mental-models-think-differently/

Gullickson, A. M., & Hannum, K. M. (2019). Making values explicit in evaluation practice. Evaluation Journal of Australasia, 19(4), 162-178. DOI: https://doi.org/10.1177/1035719X19893892

Government of Canada. (2012). Code of Values and Ethics for the Canada School of Public Service. https://www.csps-efpc.gc.ca/about_us/cve-eng.aspx

Katz, S., & Dack, L. (2013). Intentional Interrruption: breaking down Lewarning barriers to transform professional practice. SAGE Publication.

Khalid, A. F. (2019). Supporting the Use of Research Evidence to Inform Decision-Making in Crises Zones. McMaster University. https://macsphere.mcmaster.ca/bitstream/11375/25011/2/khalid_AhmadFiras_201909_PhD.pdf

Knowlton, L. S., & Phillips, C. C. (2013). The Logic Model Guidebook: Better STrategies for Great Results. Sage.

Kuji-Shikatani, K., Gallagher, M. J., Franz, R., & Börner, M. (2016). Leadership's Role in Building the Education Sector's Capacity to Use Evaluative Thinking: The Example of the Ontario Ministry of Education. In M. Patton, K. McKegg, & N. Wehipeihana, Developmental Evaluation Exemplars. The Guildford Press.

Kuji-Shikatani, K., Rowe, W., & Borner, M. (2020). Developmental Evaluation in Action: Facilitating Social Innovation in Real-Time. Canadian Evaluation Society Ontario Chapter.

Kuji-Shikatani, K., Rowe, W., & Campbell-Patton, C. (2021). Learning as we go in becoming part of the solution as a Blue Marble Evaluator (BME) in the public sector. Canadian Evaluation Society Conference 2022.

Kuji-Shikatani, K., Thompson, M., & Mathew, M. (2014). Launching the Credentialed Evaluator Designation. (H. Buchanan, & K. Kuji-Shikatani, Eds.) Canadian Journal of Program Evaluation, 29, 70-85. https://evaluationcanada.ca/system/files/cjpe-entries/29-3-070.pdf DOI: https://doi.org/10.3138/cjpe.29.3.70

Lelieveld, J. (2021). Paul J. Crutzen: Ozone Nobel prizewinner who coined the term Anthropocene. Nature. https://media.nature.com/original/magazine-assets/d41586-021-00479-0/d41586-021-00479-0.pdf

Lessard, S. (2018). Four Directions First Nations, Métis and Inuit Graduation Coach Approach. Indspire. https://indspire.ca/wp-content/uploads/2019/10/FourDirections-FINAL.pdf

McCalman, J., Jongen, C. & Bainbridge, R. Organisational systems’ approaches to improving cultural competence in healthcare: a systematic scoping review of the literature. Int J Equity Health 16, 78 (2017). https://doi.org/10.1186/s12939-017-0571-5 DOI: https://doi.org/10.1186/s12939-017-0571-5

O'Connell, M., & O'Keefe, D. T. (2021, August 5). Contact Tracking for Covid-19 - A Digital Inoculation Against Future Pandemics. doi:10.1056/NEJMp2102256 DOI: https://doi.org/10.1056/NEJMp2102256

OECD. (2016). The Governance of Inclusive Growth: An Overview of Country Initiatives. Paris: OECD Publishing. https://doi.org/10.1787/9789264265189-en DOI: https://doi.org/10.1787/9789264265189-en

OECD. (2021). The State of School Education: One Year into the COVID Pandemic. OECD Publishing: https://doi.org/10.1787/201dde84-en DOI: https://doi.org/10.1787/201dde84-en

Oxford Coronavirus Government Response Tracker. (2022). Stringency Index. Our World in Data: https://ourworldindata.org/covid-stringency-index

Oxford Coronavisrus Government Response Tracker. (2022). Containment and Health Index. Our World in Data: https://ourworldindata.org/covid-stringency-index

Patton, M. (2013, April). Making evaluation meaningful and useful. Minnesota Council on Foundations.

Patton, M.Q. (2017) Prinicples-Focused Evaluation; The GUIDE. Guilford.

Patton, M. Q. (2020). Blue Marble Evaluation - Premises and Principles . Guilford. DOI: https://doi.org/10.31244/zfe.2020.02.09

Patton, M. Q. (2021). How far dare an evaluator go toward saving the world?: Redux, update, and a reflective practice facilitation tool. American Journal of Evaluation, 42(2), 162-184. DOI: https://doi.org/10.1177/1098214020927095

Patton, M., McKegg, K., & Wehipeihana, N. (2016). Developmental evaluation exemplars: Principles in practice. New York: Guilford.

Patton, M.W., and Campbell-Patton, C.E. (2022). Utilization-Focused Evaluatuion, 5th Edition. Sage.

Ritchie, H., Mathieu, E., Rodes-Guirao, L., Appel, C., Giattino, C., Ortiz-Ospina, E., . Roser, M. (2022). Coronavirus Pandemic (Covid-19). Our World in Data: https://ourworldindata.org/coronavirus#explore-the-global-situation

Rowe, W., Chamorro, P, Hayes, G. & Corak, L. (2022). Collective Leadership and Its Contribution to Community Resiliency in Salinas, Ecuador, in S. Dhiman et al. (eds.), Handbook of Global Leadership and Followership, Springer Nature Switzerland DOI: https://doi.org/10.1007/978-3-030-75831-8_44-1

SAMEA. (2020). South African Monitoring and Evaluation Association’s Evaluator Competencies Framework. South African Monitoring and Evaluation Association.

Senge, P., Hamilton, H., & Kania, J. (2015). The dawn of system leadership. Stanford Social Innovation Review, 13(1), 27–33

Scheleicher, A. (2020). The Impact of Covid-19 on Education - Insights from Education at a Glance 2020. OECD. https://www.oecd.org/education/the-impact-of-covid-19-on-education-insights-education-at-a-glance-2020.pdf

Serrat, O. (2017). Understanding and developing emotional intelligence. Knowledge Solutions, Singapore, 329-339 DOI: https://doi.org/10.1007/978-981-10-0983-9_37

Snowden, D. J., & Boone, M. E. (2007). A leader's framework for decision making. Harvard business review, 85(11), 68.

Stevahn, L., King, J., Ghere, G., & Minnema, J. (2005). Establishing essential competencies for program evaluators. Americal Journal of Evaluation, 26(2), 43-59. https://journals.sagepub.com/doi/abs/10.1177/1098214004273180 DOI: https://doi.org/10.1177/1098214004273180

Surdeck, S. (2017, August 31). Why Understanding Other Perspectives Is A Key Leadership Skill.

Taut, S. (2008). What have we learned about stakeholder involvement in program evaluation?. Studies in Educational Evaluation, 34(4), 224-230. DOI: https://doi.org/10.1016/j.stueduc.2008.10.007

United Nations Evaluation Group (UNEG). (2016) Evaluation Competency Framework. http://www.unevaluation.org/document/detail/1915

UNESCO, UNICEF, World Bank. (2021). The State of the Global Education Crises: A Path to Recovery. https://www.unicef.org/media/111621/file/TheStateoftheGlobalEducationCrisis.pdf.pdf

UNICEF. (2020, June). Framework for reopening schools. Unicef: https://www.unicef.org/media/105286/file/Framework%20for%20Reopening%20Schools.pdf

Walinga, J., & Rowe, W. (2013). Transforming stress in complex work environments. International Journal of Workplace Health Management,, 6(1), 66-88. DOI: https://doi.org/10.1108/17538351311312420

Williams, B. (2014). System Thinking. Better Evaluation: https://www.betterevaluation.org/en/blog/systems_thinking

World Savvy.(n.d.) Global Competence Matrix. https://www.worldsavvy.org/our-approach/global-competence/

Zieba, K. (n.d.). How can systems thinking hulp us in the Covid-19 crises? Knowledge and Process Management. doi: 10.1002/kpm.1680. PMCID: PMC8441870. DOI: https://doi.org/10.1002/kpm.1680

Article Sidebar

Main Article Content