What, How, and Why? A Comparative Analysis of 12 Goal-Free Evaluations
Main Article Content
Abstract
Background: Goal-free evaluation (GFE) is any evaluation in which the evaluator conducts the evaluation without reference to predetermined goals or objectives.
Purpose: The purpose of this study is to examine GFE in actual practice focusing on what GFE is, how it is conducted, and why the evaluators use it.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Document analysis.
Data Collection and Analysis: The researcher collected data from a non-random sample of 12 cases of GFE found in published and non-published papers, reports, and guidebooks. The researcher analyzed the documents using quantitative and qualitative content analysis.
Findings: The findings suggest that goal-free evaluators consider GFE an outcome evaluation that supplements GBE. These goal-free evaluators typically used an ex post facto evaluation design, non-random sampling of stakeholders, and semi-structured interviewing to collect data. The evaluators described using GFE to improve the evaluand, to find side-effects, and to evaluate highly complex evaluands.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Alkin, M. C. (1974). Comments by Marvin C. Alkin: Wider context goals and goal-based evaluators. In W. J. Popham (Ed.), Evaluation in education: Current applications (pp. 52-55). Berkeley, CA: McCutchan.
Alkin, M. C. (2004). Evaluation roots: Tracing theorists' views and influence. Thousand Oaks, CA: Sage. https://doi.org/10.4135/9781412984157 DOI: https://doi.org/10.4135/9781412984157
Belanger, K. H. (2006, November). Goal-based vs. goal-free evaluation in crises: Lessons from 2 hurricanes. Paper presented at the meeting of the American Evaluation Association, Portland, OR.
Berkshire, S., Kouame, J., & Richardson, K. K. (2009). Making It Work: Evaluation report. Kalamazoo, MI: Western Michigan University, The Evaluation Center.
Babbie, E. (1999). The basics of social research. Belmont, CA: Wadsworth Publishers.
Brusset, E., de Coning, C., & Hughes, B. (Eds.). (2016). Complexity thinking for peacebuilding practice and evaluation. London, UK: Palgrave Macmillan. https://doi.org/10.1057/978-1-137-60111-7 DOI: https://doi.org/10.1057/978-1-137-60111-7
Bry, B. H. & George, F. E. (1979). Evaluating and improving prevention programs: A strategy from drug abuse. Evaluation and Program Planning, 2, 127-136. https://doi.org/10.1016/0149-7189(79)90022-3 DOI: https://doi.org/10.1016/0149-7189(79)90022-3
Chatterji, M. (2007). Grades of evidence: Variability in quality of findings in effectiveness studies of complex field interventions. American Journal of Evaluation, 28(3), 239-255. https://doi.org/10.1177/1098214007304884 DOI: https://doi.org/10.1177/1098214007304884
Copestake, J. G. (2014). Full guidelines for the Qualitative Impact Protocol (QUIP) (Working Paper). Retrieved from Centre for Development Studies, University of Bath: http://opus.bath.ac.uk/36678/5/QUIP_Guidelines_Oct_14.pdf
Copestake, J. & Remnant, F. (2015). The qualitative impact assessment protocol (QUIP) (Briefing Paper). Retrieved from University of Bath Centre for Development Studies website: http://www.bath.ac.uk/cds/projects-activities/qualitative-impact-assessment-protocol-QUIPS11/index.html
Cronholm, S. & Goldkuhl, G. (2003, September). Six generic types of information systems evaluation. Paper accepted to the 10th European Conference on Information Technology Evaluation, Madrid, Spain.
Davies, R. & Dart, J. (2003). A dialogical, story-based evaluation tool: The Most Significant Change technique. American Journal of Evaluation, 24(2), 127-155. https://doi.org/10.1177/109821400302400202 DOI: https://doi.org/10.1177/109821400302400202
Davies, R. & Dart, J. (2005). The 'Most Significant Change' (MSC) technique: A guide to its use. Retrieved from http://www.mande.co.uk/docs/MSCGuide.pdf
Dietz, T. et al. (2013). PADev guidebook: Participatory assessment of development. Leiden: KIT Publishers.
Dietz, T., van der Geest, K. & Obeng, F. (2013). Local perceptions of development and change in Northern Ghana. In J. Yaro (Ed.), Rural development in Northern Ghana (pp. 17-36). New York: Nova Science Publishers.
Evers, J. W. (1980). A field study of goal-based and goal-free evaluation techniques. Unpublished doctoral dissertation, Western Michigan University, Kalamazoo.
Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2004). Program evaluation: Alternative approaches and practical guidelines (3rd ed.). Boston: Pearson Education.
Hellström, T. & Jacob, M. (2003). Knowledge without goals? Evaluation of knowledge management programs. Evaluation, 9(1), 55-72. https://doi.org/10.1177/1356389003009001004 DOI: https://doi.org/10.1177/1356389003009001004
Henry, G. T. & Mark, M. M. (2003). Toward an agenda for research on evaluation. New Directions for Evaluation, 97, 69-80. https://doi.org/10.1002/ev.77 DOI: https://doi.org/10.1002/ev.77
House, E. (1991). Confessions of a responsive, goal-free evaluator. Evaluation Practice, 12(1), 109-114. https://doi.org/10.1177/109821409101200118 DOI: https://doi.org/10.1177/109821409101200118
Irving, J. F. (1979). Goal-free evaluation: Philosophical and ethical aspects of Michael Scriven's model. CEDR quarterly, 12(3), 11-14.
James, C. & Roffe, I. (2000). The evaluation of goal and goal-free training innovation. Journal of European Industrial Training, 24(1), 12-20. https://doi.org/10.1108/03090590010308228 DOI: https://doi.org/10.1108/03090590010308228
Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards: How to assess evaluations of educational programs (2nd ed.). Thousand Oaks, CA: Sage.
Lennie, J. (2011). The Most Significant Change technique: A manual of M&E staff and others at Equal Access. Washington, D.C.: USAID.
Manfredi, T. C. (2003). Goal based or goal free evaluation? Growing New Farmers website. http://www.smallfarm.org/uploads/uploads/Files/Goal_Based_or_Goal_Free_Evaluation.pdf
Matsunaga, D. S. & Enos, R. (1997). Goal-free evaluation of the Alger Foundation's Wai'anae Self-help Housing Project: Evaluation report. Kalamazoo, MI: Western Michigan University, The Evaluation Center.
Mertens, D. M. & Wilson, A. T. (2012). Program evaluation theory and practice. New York, NY: The Guilford Press.
Moro, G., Cassibba, R., & Costantini, A. (2007). Focus groups as an instrument to define evaluation criteria. Evaluation, 13(3), 340-357. https://doi.org/10.1177/1356389007078626 DOI: https://doi.org/10.1177/1356389007078626
Mueller, G. W. & Colley, L. M. (2015). An evaluation of the impact of end-of-course exams and ACT-QualityCore on U.S. history instruction in a Kentucky high school. Journal of Social Studies Research, 39, 95-106. https://doi.org/10.1016/j.jssr.2014.07.002 DOI: https://doi.org/10.1016/j.jssr.2014.07.002
Patton, M. Q. (1997). Utilization-focused evaluation: The new century text (3rd ed.). Thousand Oaks, CA: Sage Publications.
Perry, J. & Bontinen, K. (2001). Evaluation of a weekend respite program of persons with Alzheimer disease. Canadian Journal of Nursing Research, 33(1), 81-85.
Pipia, E. (2014). The effectiveness of goal-free evaluation in curriculum development and in quality assurance process. Journal of Education, 3(2), 29-31.
Rogers, P. (2011). Implications of complicated and complex characteristics for key tasks in evaluation. In K. Forss, M. Marra, and R. Schwartz (Eds.), Evaluating the complex: Attribution, contribution, and beyond. New Brunswick, New Jersey: Transaction Publishers.
Salasin, S. (1974). Exploring goal-free evaluation: An interview with Michael Scriven. Evaluation, 2(1), 9-16.
Scriven, M. (1972). Prose and cons about goal-free evaluation. Evaluation Comment, 3(4), 1-4.
Scriven, M. (1973). Goal-free evaluation. In E. R. House (Ed.), School evaluation: The politics and process (pp. 319-328). Berkeley, CA: McCutchan.
Scriven, M. (1974). Prose and cons about goal-free evaluation. In W. J. Popham (Ed.), Evaluation in education: Current applications (pp. 34-67). Berkeley, CA: McCutchan.
Scriven, M. (1991). Evaluation thesaurus (4th ed.). Newbury Park, CA: Sage.
Scriven, M. (2007, February). Key evaluation checklist. Retrieved from https://wmich.edu/sites/default/files/attachments/u350/2014/key%20evaluation%20checklist.pdf
Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation. Newbury Park, CA: Sage.
Smith, C. (2009). The Scottish enlightenment, unintended consequences and the science of man. The Journal of Scottish Philosophy, 7(1), 9-28. https://doi.org/10.3366/E1479665108000304 DOI: https://doi.org/10.3366/E1479665108000304
Stufflebeam, D. L. (1974). Comment by Daniel L. Stufflebeam: Should or can evaluation be goal-free? In W. J. Popham (Ed.), Evaluation in education: Current applications (pp. 43-46). Berkeley, CA: McCutchan.
Stufflebeam, D. L., Gullickson, A., & Wingate, L. (2002). The spirit of Consuelo: An evaluation of Ke Aka Ho'ona. Retrieved from Western Michigan University Evaluation Center website: www.wmiche.du/evalctr
Suchman, E. A. (1967). Evaluative research: Principles and practice in public service and social action programs. New York: Russell Sage Foundation.
Swisher, J. D. & Abrams, A. (1973). Specifying objectives. In L. A. Abrams, E. F. Garfield, & J. D. Swisher (Eds.), Accountability in drug education. Washington, D. C.: Drug Abuse Council.
The Oral History Project Team, O. H. P. T., & Miller, R. L. (2005). The oral history of evaluation, part 3: The professional evolution of Michael Scriven. American Journal of Evaluation, 26(3), 378-388. https://doi.org/10.1177/1098214005276646 DOI: https://doi.org/10.1177/1098214005276646
Thiagarajan, S. (1975). Goal-free evaluation of media. Educational Technology, 15(5), 38-40.
United States Agency for International Development. (2013). Discussion note: Complexity-aware monitoring. Retrieved from: http://usaidprojectstarter.org/sites/default/files/resources/pdfs/Complexity-Aware-Monitoring-2013-12-11-FINAL.pdf
Welch, W. W. "Goal Free Evaluation Report for St. Mary's Junior College" (Unpublished Report) Minneapolis: 1976.
Welch, W. W. (1978, March). Goal-free formative evaluation-an example. Paper presented at the meeting of the American Education Research Association, Toronto, Canada.
Whiting, L. S. & Roberts, S. A. (2016). "We want to influence": An evaluation of the nation health service England youth forum [Abstract]. Archives of Disease in Childhood, 101, A365-A366. https://doi.org/10.1136/archdischild-2016-310863.601 DOI: https://doi.org/10.1136/archdischild-2016-310863.601
Wilson-Grau, R. & Britt, H. (2012). Outcome harvesting. Retrieved from http://www.managingforimpact.org/sites/default/files/resource/outome_harvesting_brief_final_2012-05-2-1.pdf
World Bank. (2014). Cases in outcome harvesting: Ten pilot experiences identify new learning from multi-stakeholder project to improve results. Washington, D.C.: The World Bank. Retrieved from: http://documents.worldbank.org/curated/en/419021468330946583/pdf/901720WP0Box380n0Outcome0Harvesting.pdf
Worthen, B. (1990). Program evaluation. In H. Walberg & G. Haertel (Eds.), The international encyclopedia of educational evaluation (pp. 42-47). Toronto, ON: Pergammon Press.
Youker, B. W. (2005). Goal-free evaluation of the 2005 Kalamazoo Public School Middle School Summer Enrichment Program: Final report. Kalamazoo, MI: Western Michigan University, The Evaluation Center.
Youker, B. W. (2013). Goal-free evaluation: A potential model for the evaluation of social work programs. Social Work Research, 37(4), 432-438. https://doi.org/10.1093/swr/svt032 DOI: https://doi.org/10.1093/swr/svt032
Youker, B. W., Ford, K., & Bayer, N. (2017). Dismissing the goals: A comparison of four goal-dismissive goal-free evaluations. Evaluation and Program Planning, 46, 1-6. https://doi.org/10.1016/j.evalprogplan.2017.05.007 DOI: https://doi.org/10.1016/j.evalprogplan.2017.05.007
Youker, B. W., Ingraham, A., & Bayer, N. (2014). An assessment of goal-free evaluation: Case studies of four goal-free evaluations. Evaluation and Program Planning, 46, 10-16. https://doi.org/10.1016/j.evalprogplan.2014.05.002 DOI: https://doi.org/10.1016/j.evalprogplan.2014.05.002
Youker, B. W., Zielinski, A., Hunter, O. C. & Bayer, N. (2016). Who needs goals? A case study of goal-free evaluation. Journal of Multidisciplinary Evaluation, 12(27), 27-43. https://doi.org/10.56645/jmde.v12i27.448 DOI: https://doi.org/10.56645/jmde.v12i27.448