Investigating Equity and Social Justice within Programs: A Framework of Evaluative Criteria
Article Sidebar
Main Article Content
Abstract
Background: Evaluators have a professional and ethical responsibility to contribute to the “advancement of an equitable and just society.” A rich body of scholarship provides guidance about how evaluators can do so through social justice–oriented and decolonizing evaluation approaches, culturally responsive methods, attending to power and privilege in program and evaluation contexts, and partnering with communities. In this article, we provide guidance for examining how the program being evaluated attends to issues of equity and social justice.
Purpose: We present a framework for investigating equity and social justice within programs through the criteria that evaluations pursue. The framework is offered as a map of possibilities and a thinking tool to help evaluators surface, examine, and negotiate varying values and design evaluative lines of inquiry to address them.
Setting: We write as evaluation faculty in colleges of education. We approached the framework through the lens of our shared commitment to advancing equity and social justice in evaluation practice. We aimed to bring together our differing areas of expertise and lived experiences to develop a resource to support evaluators in advancing equity and social justice.
Data Collection and Analysis: Our work builds on a previously published model of criteria domains. Each domain reflects a broad category of program characteristics or results that can be deemed important or desirable for a given program and context. Here, we refine and expand the model by applying an equity and social justice lens to 11 different criteria domains. We draw on the social justice–oriented evaluation literature for guidance and examples about how evaluators can advance equity and social justice within each domain and associated lines of inquiry.
Findings: The framework outlines 11 criteria domains in which a program’s contribution to equity and social justice might be examined. We describe each domain and apply it to an example evaluation to illustrate. We conclude by discussing the use of the framework to advance equity and social justice through evaluation practice, education, and scholarship.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
American Evaluation Association. (2018a). AEA evaluator competencies. https://www.eval.org/About/Competencies-Standards/AEA-Evaluator-Competencies
American Evaluation Association. (2018b). Guiding principles for evaluators. https://www.eval.org/About/Guiding-Principles
American Evaluation Association. (n.d.). Dialogues on race and class. https://www.eval.org/Events/Dialogues-on-Race-and-Class
American Medical Association. (2021). Advancing health equity: A guide to language, narrative and concepts. https://www.ama-assn.org/about/ama-center-health-equity/advancing-health-equity-guide-language-narrative-and-concepts-0
Armytage, L. (2011). Evaluating aid: An adolescent domain of practice. Evaluation, 17(3), 261-276. https://doi.org/10.1177/1356389011410518
Augustsson, H., Richter, A., Hasson, H., & von Thiele Schwarz, U. (2017). The need for dual openness to change: A longitudinal study evaluating the impact of employees' openness to organizational change content and process on intervention outcomes. The Journal of Applied Behavioral Science, 53(3), 349-368. https://doi.org/10.1177/0021886317691930
Avent, C. M. (2025). The criticality of language: Exploring STEM education evaluators conceptualizations of equity, diversity, and inclusion and the influence on their roles and practice. Evaluation and Program Planning, 108. https://doi.org/10.1016/j.evalprogplan.2024.102511
Avent, C., Teasdale, R. M., Yan, X., Serrano Abreu, M. B., & Moore, C. (2023). Towards racial equity: Navigating diverse conceptualizations and perspectives in social justice-oriented evaluation practice [Manuscript submitted for publication].
Bamberger, M., & Segone, M. (2011). How to design and manage equity-focused evaluations. https://agora.unicef.org/course/info.php?id=1238
Benitez, M., Jr. (2010). Resituating culture centers within a social justice framework: Is there room for examining Whiteness? In L. D. Patton (Ed.), Culture centers in higher education: Perspectives on identity, theory, and practice (pp. 119-134). Stylus.
https://doi.org/10.4324/9781003443971-9
Bledsoe, K. (2021). Evaluation's identity: Who are we, why are we here, and what does our future look like? Evaluation Matters-He Take Tō Te Aromatawai, 7, 6-29. https://doi.org/10.18296/em.0064
Bowen, M. L., & Tillman, A. S. (2015). Developing culturally responsive surveys: Lessons in development, implementation, and analysis from Brazil's African descent communities. American Journal of Evaluation, 36(1), 25-41. https://doi.org/10.1177/1098214014539831
Boyce, A., & Smith, T. (2021). Diversity, equity, and inclusion in evaluation: Definitions, evaluation questions, indicators, and data collection methods. https://evalu-ate.org/guide/edi-in-evaluation/
Boyce, A. S., Tovey, T. L. S., Onwuka, O., Moller, J. R., Clark, T., & Smith, A. (2023). Exploring NSF-funded evaluators' and principal investigators' definitions and measurement of diversity, equity, and inclusion. American Journal of Evaluation, 44(1), 50-73. https://doi.org/10.1177/10982140221108662
Canadian Evaluation Society. (n.d.). 2019-2024 Canadian Evaluation Society strategic plan. https://evaluationcanada.ca/client_assets/PDFs/CES%20Strategic%20Plan%20EN.pdf
Centre for Excellence for Evaluation. (2015). Evaluation of the 2009 Policy on Evaluation. https://www.canada.ca/en/treasury-board-secretariat/services/audit-evaluation/centre-excellence-evaluation/evaluation-2009-policy-evaluation.html
Chianca, T. (2008). The OECD/DAC criteria for international development evaluations: An assessment and ideas for improvement. Journal of MultiDisciplinary Evaluation, 5(9), 41-51. https://doi.org/10.56645/jmde.v5i9.167
Chilisa, B., & Bowman, N. R. (2023). Special issue editors' introductory note: The why and how of the decolonization discourse. Journal of MultiDisciplinary Evaluation, 19(44), 2-10.
https://doi.org/10.56645/jmde.v19i44.919
Chilisa, B., Major, T. E., Goathobogwe, M., & Mokgolodi, H. (2016). Decolonizing and indigenizing evaluation practice in Africa: Toward African relational evaluation approaches. Canadian Journal of Program Evaluation, 30(3), 313-328. https://doi.org/10.3138/cjpe.30.3.05
Cram, F. (2016). Lessons on decolonizing evaluation from Kaupapa Māori evaluation. Canadian Journal of Program Evaluation, 30(3), 296-312. https://doi.org/10.3138/cjpe.30.3.04
Cronbach, L. J., & Associates. (1980). Toward reform of program evaluation. Jossey-Bass.
Davidson, E. J. (2005a). Criteria. In S. Mathison (Ed.), Encyclopedia of Evaluation (pp. 91-92). Sage.
Davidson, E. J. (2005b). Evaluation methodology basics: The nuts and bolts of sound evaluation. Sage. https://doi.org/10.4135/9781452230115
Dean-Coffey, J. (2018). What's race got to do with it? Equity and philanthropic evaluation practice. American Journal of Evaluation, 39(4), 527-542. https://doi.org/10.1177/1098214018778533
Dickinson, P., & Adams, J. (2017). Values in evaluation: The use of rubrics. Evaluation and Program Planning, 65, 113-116. https://doi.org/10.1016/j.evalprogplan.2017.07.005
Dumaine, F. (2012). When one must go: The Canadian experience with strategic review and judging program value. New Directions for Evaluation, 133, 65-75. https://doi.org/10.1002/ev.20007
Elam, P., & Walker, W. (2021). Considerations for conducting evaluation using a culturally responsive and racial equity lens. https://mphi.org/wp-content/uploads/2022/05/Considerations-for-Conducting-Evaluation-Using-a-Culturally-Responsive-and-Racial-Equity-Lens.pdf
Fournier, D. M. (1995). Establishing evaluative conclusions: A distinction between general and working logic. New Directions for Evaluation, 68, 15-32. https://doi.org/10.1002/ev.1017
Garibay, C., & Teasdale, R. M. (2019). Equity and evaluation in informal STEM education. New Directions for Evaluation, 161, 87-106. https://doi.org/10.1002/ev.20352
Gates, E. F., Madres, J., Hall, J. N., & Alvarez, K. B. (2022). It takes an ecosystem: Socioecological factors influencing equity-oriented evaluation in New England, U.S., 2021. Evaluation and Program Planning, 92, 102068. https://doi.org/10.1016/j.evalprogplan.2022.102068
Gates, E. F., Teasdale, R. M., Shim, C., & Hubacz, H. (2024). Whose and what values? Advancing and illustrating explicit specification of evaluative criteria in education. Studies in Educational Evaluation, 81, 101335. https://doi.org/10.1016/j.stueduc.2024.101335
Gates, E. F., Williamson, E., Madres, J., Hall, J. N., & Alvarez, K. B. (2022). Equity "on the sideline": A mixed methods study of New England evaluation practice in 2020. Journal of MultiDisciplinary Evaluation, 18(42). https://doi.org/10.56645/jmde.v18i42.715
Giacomini, M., & Hurley, J. (2008). Issues in evaluating equity. In L. Potvin, D. V. McQueen, M. Hall, L. de Salazar, L. M. Anderson, & Z. M. Hartz (Eds.), Health promotion evaluation practices in the Americas (pp. 285-298). Springer.
https://doi.org/10.1007/978-0-387-79733-5_16
Greene, J. C. (2006). Evaluation, democracy, and social change. In I. Shaw, J. C. Greene, & M. M. Mark (Eds.), The SAGE Handbook of Evaluation (pp. 118-140). Sage.
https://doi.org/10.4135/9781848608078.n5
Greene, J. C. (2012). Values-engaged evaluations. In M. Segone (Ed.), Evaluation for equitable development results (pp. 192-206). UNICEF. http://www.clear-la.cide.edu/sites/default/files/Evaluation_for_equitable%20results_web.pdf
Greene, J. C., Boyce, A. S., & Ahn, J. (2011). A values-engaged, educative approach for evaluating education programs: A guidebook for practice. University of Illinois at Urbana-Champaign. http://comm.eval.org/communities/community-home/librarydocuments/viewdocument?DocumentKey=f3c734c0-8166-4ba4-9808-a07e05294583
Gullickson, A. M. (2020). The whole elephant: Defining evaluation. Evaluation and Program Planning, 79, 101787. https://doi.org/10.1016/j.evalprogplan.2020.101787
Gullickson, A. M., & Hannum, K. M. (2019). Making values explicit in evaluation practice. Evaluation Journal of Australasia, 19(4), 162-178. https://doi.10.1177/1035719X19893892
Hall, J. N. (2020a). Focus groups: Culturally responsive approaches for qualitative inquiry and program evaluation. Myers Education.
Hall, J. N. (2020b). The other side of inequality: Using standpoint theories to examine the privilege of the evaluation profession and individual evaluators. American Journal of Evaluation, 41(1), 20-33. https://doi.org/10.1177/1098214019828485
Hall, M. E. (2018). Evaluation's race problem in the U.S.: A call to action for the profession and the American Journal of Evaluation. American Journal of Evaluation, 39(4), 569-583. https://doi.org/10.1177/1098214018792624
Hall, M. E. (2020). Blest be the ties that bind. New Directions for Evaluation, 166, 13-22. https://doi.org/10.1002/ev.20414
Hood, S. (2001). Nobody knows my name: In praise of African American evaluators who were responsive. New Directions for Evaluation, 92, 31-44. https://doi.org/10.1002/ev.33
Hood, D. W., & Cassaro, D. A. (2002). Feminist evaluation and the inclusion of difference. New Directions for Evaluation, 96, 27-40. https://doi.org/10.1002/ev.64
Hood, S., Hopson, R. K., & Kirkhart, K. E. (2015). Culturally responsive evaluation: Theory, practice, and future implications. In K. E. Newcomer, H. P. Hatry, & J. S. Wholey (Eds.), Handbook of practical program evaluation (4th ed., pp. 281-317). Jossey-Bass.
https://doi.org/10.1002/9781119171386.ch12
Hopson, R., & Cram, F. (2018). Tackling wicked problems in complex evaluation ecologies. In R. Hopson & F. Cram (Eds.), Tackling wicked problems in complex ecologies: The role of evaluation (pp. 3-24). Stanford University Press. https://doi.org/10.1515/9781503605565
House, E. R. (2015). Evaluating: Values, biases, and practical wisdom. Information Age.
House, E. R., & Howe, K. R. (1999). Values in evaluation and social research. Sage.
https://doi.org/10.4135/9781452243252
Hurteau, M., Houle, S., & Mongiat, S. (2009). How legitimate and justified are judgments in program evaluation?. Evaluation, 15(3), 307-319. https://doi.org/10.1177/1356389009105883
Jacobson, M. R., Azzam, T., & Baez, J. (2012). The nature and frequency of inclusion of people with disabilities in program evaluation. American Journal of Evaluation, 34(1), 23-44. https://doi.org/10.1177/1098214012461558
Jordan, L. S., & Hall, J. N. (2023). Framing anticolonialism in evaluation: Bridging decolonizing methodologies and culturally responsive evaluation. Journal of MultiDisciplinary Evaluation, 19(44), 102-116. https://doi.org/10.56645/jmde.v19i44.769
King, J., McKegg, K., Oakden, J., & Wehipeihana, N. (2013). Evaluative rubrics: A method for surfacing values and improving the credibility of evaluation. Journal of MultiDisciplinary Evaluation, 9(21), 11-20. https://doi.org/10.56645/jmde.v9i21.374
Kirkhart, K. E. (2016). Equity, privilege, and validity: Traveling companions or strange bedfellows? In S. I. Donaldson & R. Picciotto (Eds.), Evaluation for an equitable society (pp. 109-131). Information Age.
Madison, A. (1992). Primary inclusion of culturally diverse minority program participants in the evaluation process. New Directions for Evaluation, 53, 35-43. https://doi.org/10.1002/ev.1599
Madison, A. M. (2000). Language in defining social problems and in evaluating social programs. New Directions for Evaluation, 86, 17-28. https://doi.org/10.1002/ev.1169
Manswell-Butty, J. L., Reid, M. D., LaPoint, V. (2004). A culturally responsive evaluation approach applied to the talent development school-to-career intervention program. New Directions for Evaluation, 101, 37-47. https://doi.org/10.1002/ev.106
Mark, M. M., Henry, G. T., & Julnes, G. (2000). Evaluation: An integrated framework for understanding, guiding, and improving policies and programs. Jossey-Bass.
Martens, K. S. R. (2018). Rubrics in program evaluation. Evaluation Journal of Australasia, 18(1), 21-44. https://doi.org/10.1177/1035719X17753961
McBride, D., Casillas, W., & LoPiccolo, J. (2020). Inciting social change through evaluation. New Directions for Evaluation, 166, 119-127. https://doi.org/10.1002/ev.20405
Mertens, D. M. (1999). Inclusive evaluation: Implications of transformative theory for evaluation. American Journal of Evaluation, 20(1), 1-14. https://doi.org/10.1177/109821409902000102
Mertens, D. M., & Hopson, R. K. (2006). Advancing evaluation of STEM efforts through attention to diversity and culture. New Directions for Evaluation, 109, 35-51. https://doi.org/10.1002/ev.177
Mertens, D. M., & Wilson, A. T. (2019). Program evaluation theory and practice: A comprehensive guide (2nd ed.). Guilford.
Miller, R. L. (2018). Hiding in plain sight: On culturally responsive evaluation and LGBT communities of colour. Evaluation Matters-He Take Tō Te Aromatawai, 4, 5-34. https://doi.org/10.18296/em.0027
Miller, V. D., Johnson, J. R., & Grau, J. (1994). Antecedents to willingness to participate in a planned organizational change. Journal of Applied Communication Research, 22, 59-80. https://doi.org/10.1080/00909889409365387
Montrosse-Moorhead, B. (2022, October 13). Evaluation synthesis: What is it? How do we teach it to others? [Video]. https://vimeo.com/showcase/10952695/video/762437455
National Academies of Sciences, Engineering, and Medicine. (2017). Undergraduate research experiences for STEM students: Successes, challenges, and opportunities. National Academies Press. https://doi.org/https://doi.org/10.17226/24622
Neubauer, L. C., McBride, D., Guajardo, A. D., Casillas, W. D., & Hall, M. E. (2020). Examining issues facing communities of color today: The role of evaluation to incite change. New Directions for Evaluation, 166, 7-11. https://doi.org/10.1002/ev.20406
Organisation for Economic Co-operation and Development. (1991). Principles for evaluation of development assistance. http://www.oecd.org/dac/evaluation/50584880.pdf
Organisation for Economic Co-operation and Development. (1999). Guidance for evaluating humanitarian assistance in complex emergencies. https://www.oecd.org/dac/evaluation/2667294.pdf
Organisation for Economic Co-operation and Development. (2002). Glossary of key terms in evaluation and results based management. http://www.oecd.org/dac/evaluation/2754804.pdf
Organisation for Economic Co-operation and Development. (2018). DAC criteria for evaluating development assistance. http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm
Organisation for Economic Co-operation and Development. (2019). Better criteria for better evaluation: Revised evaluation criteria definitions and principles for use. https://www.oecd.org/dac/evaluation/revised-evaluation-criteria-dec-2019.pdf
Ozeki, S., Coryn, C. L. S., & Schröter, D. C. (2019). Evaluation logic in practice: Findings from two empirical investigations of American Evaluation Association members. Evaluation and Program Planning, 76, 101681. https://doi.org/10.1016/j.evalprogplan.2019.101681
Patton, M. Q. (2021). Evaluation criteria for evaluating transformation: Implications for the coronavirus pandemic and the global climate emergency. American Journal of Evaluation, 42(1), 53-89. https://doi.org/10.1177/1098214020933689
Phillips, G. I., Felt, D., Perez-Bill, E., Ruprecht, M. M., Glenn, E. E., Linderman, P., & Miller, R. L. (2023). Transforming the paradigm for LGBTQ+ evaluation: Advancing a praxis of LGBTQ+ inclusion and liberation in evaluation. American Journal of Evaluation, 44(1), 7-28. https://doi.org/10.1177/10982140211067206
Rogers, P. J. (2016). Understanding and supporting equity: Implications of methodological and procedural choices in equity-focused evaluations. In S. I. Donaldson & R. Picciotto (Eds.), Evaluation for an equitable society (pp. 199-216). Information Age.
Roorda, M., & Gullickson, A. M. (2019). Developing evaluation criteria using an ethical lens. Evaluation Journal of Australasia, 19(4), 179-194. https://doi.org/10.1177/1035719X19891991
Rossi, P. H., Lipsey, M. W., & Henry, G. T. (2019). Evaluation: A systematic approach (8th ed.). Sage.
Sadler, D. R. (1985). The origins and functions of evaluative criteria. Educational Theory, 35(3), 285-297. https://doi.org/10.1111/j.1741-5446.1985.00285.x
Schwandt, T. A. (2015). Evaluation foundations revisited: Cultivating a life of the mind for practice. Stanford University Press.
Schwandt, T. A., & Gates, E. F. (2021). Evaluating and valuing in social research. Guilford.
Scriven, M. (2000). Evaluation ideologies. In D. L. Stufflebeam, G. F. Madaus, & T. Kellaghan (Eds.), Evaluation models: Viewpoints on educational and human services evaluation (2nd ed., pp. 249-278). Kluwer. https://doi.org/10.1007/0-306-47559-6_15
Scriven, M. (2007). The logic of evaluation. In H. V. Hansen, C. W. Tindale, J. A. Blair, & R. H. Johnson (Eds.), Proceedings of the 7th Ontario Society for the Study of Argumentation Conference. https://scholar.uwindsor.ca/ossaarchive/OSSA7/papersandcommentaries/138
Scriven, M. (2012). The logic of valuing. New Directions for Evaluation, 133, 17-28. https://doi.org/10.1002/ev.20003
Segone, M. (2012). Evaluation to accelerate progress towards equity, social justice and human rights. In M. Segone (Ed.), Evaluation for equitable development results (pp. 2-12). UNICEF. http://www.clear-la.cide.edu/sites/default/files/Evaluation_for_equitable%20results_web.pdf
Shipman, S. (2012). The role of context in valuing federal programs. New Directions for Evaluation, 133, 53-63. https://doi.org/10.1002/ev.20006
Sielbeck-Bowen, K. A., Brisolara, S., Seigart, D., Tischler, C., & Whitmore, E. (2002). Exploring feminist evaluation: The ground from which we rise. New Directions for Evaluation, 96, 3-8. https://doi.org/10.1002/ev.62
Stake, R., Migotsky, C., Davis, R., Cisneros, E. J., Depaul, G., Dunbar, C., Farmer, R., Feltovich, J., Johnson, E., Williams, B., Zuria, M., & Chaves, I. (1997). The evolving syntheses of program value. Evaluation Practice, 18(2), 89. https://doi.org/10.1016/S0886-1633(97)90015-5
Teasdale, R. M. (2021). Evaluative criteria: An integrated model of domains and sources. American Journal of Evaluation, 42(3), 354-376. https://doi.org/10.1177/1098214020955226
Teasdale, R. M. (2022a). How do you define success? Evaluative criteria for informal STEM education. Visitor Studies, 25(2), 163-184. https://doi.org/10.1080/10645578.2022.2056397
Teasdale, R. M. (2022b). Representing the values of program participants: Endogenous evaluative criteria. Evaluation and Program Planning, 94, 102123. https://doi.org/10.1016/j.evalprogplan.2022.102123
Teasdale, R. M., Avent, C. M., Moore, C., Serrano Abreu, M. B., & Yan, X. (2024). "It has to be the common thread": Weaving attention to racial equity and justice across lines of inquiry and evaluative criteria. American Journal of Evaluation. Advanced online publication. https://doi.org/10.1177/10982140241260069
Teasdale, R. M., Moore, C., & Strasser, M. (in press). Reporting lines of inquiry: Documenting evaluations and making values explicit. Canadian Journal of Program Evaluation.
Teasdale, R. M., Pitts, R. T., Gates, E. F., & Shim, C. (2023). Teaching specification of evaluative criteria: A guide for evaluation education. New Directions for Evaluation, 117, 31-37. https://doi.org/10.1002/ev.20546
Teasdale, R. M., Strasser, M., Moore, C., & Graham, K. (2023). Evaluative criteria in practice: Findings from an analysis of evaluations published in Evaluation and Program Planning. Evaluation and Program Planning, 97, 102226. https://doi.org/10.1016/j.evalprogplan.2023.102226
Thomas, V. G. (2004). Building a contextually responsive evaluation framework: Lessons from working with urban school interventions. New Directions for Evaluation 101, 3-23. https://doi.org/10.1002/ev.104
Thomas, V. G., & Campbell, P. B. (2021). Evaluation in today's world: Respecting diversity, improving quality, and promoting usability. Sage.
Thomas, V. G., Madison, A., Rockcliffe, F., DeLaine, K., & McDonald Lowe, S. (2018). Racism, social programming, and evaluation: Where do we go from here? American Journal of Evaluation, 39(4), 514-526. https://doi.org/10.1177/1098214018772910
Thomas, V. G., & Parsons, B. A. (2017). Culturally responsive evaluation meets systems-oriented evaluation. American Journal of Evaluation, 38(1), 7-28. https://doi.org/10.1177/1098214016644069
Tovey, T. L. S., & Archibald, T. (2023). The relationship between reflective practice, evaluative thinking, and practical wisdom. In M. Hurteau & T. Archibald (Eds.), Practical wisdom for an ethical evaluation practice. Information Age.
Treasury Board of Canada Secretariat. (2012). Policy on evaluation. https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15024
Treasury Board of Canada Secretariat. (2016). Policy on results. https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=31300
United States General Accounting Office. (1988). Children's programs: A comparative evaluation framework and five illustrations. https://www.gao.gov/products/136936
Waapalaneexkweew. (2018). Looking backward but moving forward: Honoring the sacred and asserting the sovereign in Indigenous evaluation. American Journal of Evaluation, 39(4), 543-568. https://doi.org/10.1177/1098214018790412
Wanberg, C. R., & Banas, J. T. (2000). Predictors and outcomes of openness to changes in a reorganizing workplace. Journal of Applied Psychology, 85(1), 132-142. https://doi.org/10.1037/0021-9010.85.1.132
Yarbrough, D. B., Shulha, L. M., & Hopson, R. K. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Sage