Developing Criteria to Identify Transformative Participatory Evaluators
Main Article Content
Abstract
Background: The evaluation discipline continues to evolve as more and more researchers study practice. The research described in this article further defines Transformative Participatory Evaluation (T-PE) by focusing on the key elements that practitioners and theorists agree define this evaluation practice.
Purpose: A multi-stage, mixed-method approach was used to develop and examine a set of statements that serve two purposes: First, they can help identify a subset of participatory practitioners from others and, second, they further theory development by showing how T-PE practitioners differ from other evaluation practitioners on key indicators.
Setting: In the first phase of this research, three prominent evaluation theorists comprised an expert panel to develop a set of statements that would identify T-PE practitioners. The American Evaluation Association membership was used to test the statements in the research’s subsequent phases.
Intervention: NA
Research Design: A multi-stage, mixed-method approach was used to develop and test the statements.
Data Collection and Analysis: The panel was engaged in a web-based wiki to jointly edit the statements; an online questionnaire with mostly closed-ended items was used to gather AEA member input; a unique online modeling software and webinars were used to further understand findings. Analysis of variance was used to assess differences between groups and Rasch modeling and Wald tests were used to analyze the modeling data.
Findings: The eight core statements that emerged had acceptable internal reliability and limited construct validity. Though the statements’ discrimination strength was tenuous, quantitative comparisons of preferred evaluation practice models showed congruence with the predicted underlying philosophies and therefore supports the statements’ ability to discern T-PE evaluators from P-PE evaluators.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Alkin, M. C. (2004). Comparing evaluation points of view. In M. C. Alkin (Ed.), Evaluation roots: Tracing theorists' views and influences (pp. 3-11). Thousand Oaks, CA: Sage. https://doi.org/10.4135/9781412984157.n1 DOI: https://doi.org/10.4135/9781412984157.n1
Brisolara, S. (1998). The history of participatory evaluation and current debates in the field. New Directions for Evaluation, 80, 25-41. https://doi.org/10.1002/ev.1115 DOI: https://doi.org/10.1002/ev.1115
Burke, B. (1998). Evaluating for a change: Reflections on participatory methodology. New Directions for Evaluation, 80, 43-56. https://doi.org/10.1002/ev.1116 DOI: https://doi.org/10.1002/ev.1116
Checkoway, B. N., & Gutierrez, L. M. (2006). Youth participation and community change: An introduction. In B. N. Checkoway & L. M. Gutierrez (Eds.), Youth participation and community change (pp. 1-9). New York, NY: The Howarth Press. https://doi.org/10.1300/J125v14n01_01 DOI: https://doi.org/10.1300/J125v14n01_01
Checkoway, B., & Richards-Schuster, K. (2003). Youth participation in community evaluation research. American Journal of Evaluation, 24(1), 21-33. https://doi.org/10.1177/109821400302400103 DOI: https://doi.org/10.1177/109821400302400103
Cousins, J. B., Whitmore, E., & Shulha, L. (2013). Arguments for a common set of principles for collaborative inquiry in evaluation. American Journal of Evaluation, 34(1), 7-22. https://doi.org/10.1177/1098214012464037 DOI: https://doi.org/10.1177/1098214012464037
Cousins, J. B., & Chouinard, J. A. (2012). Participatory evaluation up close: A review and integration of the research base. Charlotte, NC: Information Age Press. DOI: https://doi.org/10.1108/978-1-61735-803-6
Cousins, J. B., & Earl, L. M. (1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis, 14(4), 397. https://doi.org/10.3102/01623737014004397 DOI: https://doi.org/10.2307/1164283
Cousins, J. B., & Whitmore, E. (1998). Framing participatory evaluation. New Directions for Evaluation, 80, 5-23. https://doi.org/10.1002/ev.1114 DOI: https://doi.org/10.1002/ev.1114
Crano, W. D., & Brewer, M. B. (2002). Principles and methods of social research (2nd ed.). Mahwah, NJ: Lawrence Earlbaum Associates.
DeVellis, R. F. (2003). Scale development: Theory and applications (2nd ed.). Thousand Oaks, CA: Sage.
Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Earlbaum Associates.
Estrella, M., & Gaventa, J. (1997). Who counts reality? Participatory monitoring and evaluation: A literature review. International Institute for Rural Construction, 5, 1-70.
Fetterman, D. M. (2005). A window into the heart and soul of empowerment evaluation: Looking through the lens of empowerment evaluation principles. In D. M. Fetterman & A. Wandersman (Eds.), Empowerment evaluation principles in practice (pp. 1-26). New York, NY: The Guilford Press.
Fleischer, D. N., & Christie, C. A. (2009). Evaluation use: Results from a survey of U.S. American Evaluation Association members. American Journal of Evaluation, 30(2), 158-175. https://doi.org/10.1177/1098214008331009 DOI: https://doi.org/10.1177/1098214008331009
Gargani, J. (2003). A novel quantitative method for developing and analyzing program theory. Unpublished manuscript.
Gaventa, J. (1993). The powerful, the powerless, and the experts: Knowledge struggles in an information age. In P. Park, M. Brydon-Miller, B. Hall, & T. Jackson (Eds.), Voices of change: Participatory research in the United States and Canada (pp. 2-46). Toronto, Canada: OISE Press.
Greene, J. C. (1988a). Stakeholder participation and utilization in program evaluation. Evaluation Review, 12(2), 91-116. https://doi.org/10.1177/0193841X8801200201 DOI: https://doi.org/10.1177/0193841X8801200201
Greene, J. C. (1988b). Communication of results and utilization in participatory program evaluation. Evaluation and Program Planning, 11(4), 341-351. https://doi.org/10.1016/0149-7189(88)90047-X DOI: https://doi.org/10.1016/0149-7189(88)90047-X
Greene, J. C. (2007). Mixed methods in social inquiry. San Francisco, CA: Jossey-Bass.
Greenwood, D. J., Whyte, W. F., & Harkavy, I. (1993). Participatory action research as a process and as a goal. Human Relations, 46(2), 175-192. https://doi.org/10.1177/001872679304600203 DOI: https://doi.org/10.1177/001872679304600203
Harnar, M. A. (2012). Theory Building Through Praxis Discourse: A Theory- And Practice-Informed Model Of Transformative Participatory Evaluation. (Unpublished doctoral dissertation). Claremont Graduate University, Claremont, CA.
Howell, D. C. (2002). Statistical methods for psychology (5th ed.). Pacific Grove, CA: Duxbury.
King, J. A. (1998). Making sense of participatory evaluation practice. New Directions for Evaluation, 80, 57-67. https://doi.org/10.1002/ev.1117 DOI: https://doi.org/10.1002/ev.1117
Mertens, D. M. (2009). Transformative research and evaluation. New York, NY: The Guilford Press.
Meyer, L. H., Park, H., Grenot-Scheyer, M., Schwartz, I., & Harry, B. (1998). Participatory research: New approaches to the research to practice dilemma. The Journal of the Association for Persons with Severe Handicaps, 23(3), 165-177. https://doi.org/10.2511/rpsd.23.3.165 DOI: https://doi.org/10.2511/rpsd.23.3.165
Miller, R. L. (2010). Developing standards for empirical examinations of evaluation theory. American Journal of Evaluation, 31(3), 390-399. https://doi.org/10.1177/1098214010371819 DOI: https://doi.org/10.1177/1098214010371819
Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage.
Preskill, H., & Caracelli, V. (1997). Current and developing conceptions of use: Evaluation Use TIG Survey Results. American Journal of Evaluation, 18(1), 209-225. https://doi.org/10.1177/109821409701800122 DOI: https://doi.org/10.1177/109821409701800122
Sabo Flores, K. (2008). Youth participatory evaluation: Strategies for engaging young people. San Francisco, CA: Jossey-Bass.
Sabo, K. (1999). Young people's involvement in evaluating the programs that serve them. (Unpublished doctoral dissertation). City University of New York.
Shadish, W. R. (1998). Evaluation theory is who we are. American Journal of Evaluation, 19(1), 1-19. https://doi.org/10.1177/109821409801900102 DOI: https://doi.org/10.1016/S1098-2140(99)80177-5
Streiner, D. L., & Norman, G. R. (2008). Health measurement scales: A practical guide to their development and use (4th ed.). New York, NY: Oxford University Press. https://doi.org/10.1093/acprof:oso/9780199231881.001.0001 DOI: https://doi.org/10.1093/acprof:oso/9780199231881.001.0001
Suarez-Balcazar, Y., & Harper, G. W. (2003). Community-based approaches to empowerment and participatory evaluation. Journal of Prevention & Intervention in the Community, 25(3), 1-4. https://doi.org/10.1300/J005v26n02_01 DOI: https://doi.org/10.1300/J005v26n02_01
Whitmore, E. (1988). Participatory evaluation approaches: Side effects and empowerment. (Unpublished doctoral dissertation). Cornell University, Ithaca, NY.