Who Needs Goals? A Case Study of a Goal-Free Evaluation
Main Article Content
Abstract
Background: Goal-free evaluation (GFE) is any evaluation in which the evaluator conducts the evaluation without knowledge of or reference to predetermined goals and objectives whereas the goal-based evaluator determines merit according to the evaluand’s goal achievement.
Purpose: To examine a GFE in actual practice focusing on its operationalization as well as paying particular attention to the evaluation users’ perspective of its utility.
Setting: The evaluand was a day long training of summer camp counselors on occupational therapy (OT) related skills such as feeding, dressing, and bathing.
Intervention: GFE was the intervention however for comparison purposes an independent and simultaneous goal-based evaluation (GBE) also evaluated the evaluand.
Research Design: Case study.
Data Collection and Analysis: After the evaluation users read both the GBE and GFE reports, data collection consisted of a semantic differential questionnaires followed by a focus group. Additionally, the research team analyzed both GBE and GFE reports for relevant themes.
Findings: The evaluation users reported a slightly more positive attitude toward the GFE report on the semantic differential yet many focus group respondents stated that they found the GBE report more useful or perceived no difference between the two. Evaluation users reported the benefits of GFE to include its potential for developing or aligning goals, expanding the pool of potential outcomes, and supplementing GBE strategies.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Alkin, M. C. (1972). Wider context goals and goal-based evaluators. Evaluation Comment: The Journal of Educational Evaluation, 3(4), 10-11.
Alkin, M. C. (2004). Comparing evaluation points of view. In M. Alkin (Ed.), Evaluation roots: Tracing theorists' views and influence (pp. 3-11). Thousand Oaks, CA: Sage. https://doi.org/10.4135/9781412984157.n1 DOI: https://doi.org/10.4135/9781412984157.n1
Berkshire, S., Kouame, J., & Richardson, K. K. (2009). Making It Work: Evaluation report. Kalamazoo, MI: Western Michigan University, The Evaluation Center.
Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: Handbook I. Cognitive domain. New York: David McKay.
Campbell, S., & Stanley, J. C. (1963/1966). Experimental and quasi-experimental designs for research. Boston: Houghton Mifflin.
Chen, H., & Rossi, P. (1983). Evaluating with sense: The theory driven approach. Evaluation Review, 7, 283-302. https://doi.org/10.1177/0193841X8300700301 DOI: https://doi.org/10.1177/0193841X8300700301
Christie, C. A. (2012). Advancing empirical scholarship to further develop evaluation theory and practice. The Canadian Journal of Program Evaluation, 26(1), 1-18. https://doi.org/10.3138/cjpe.026.001 DOI: https://doi.org/10.3138/cjpe.026.001
Cook, T., & Campbell, D. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally.
Coryn, C. L. S., Noakes, L. A., Westine, C. D., & Schröter, D. C., (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32(2), 199-226. https://doi.org/10.1177/1098214010389321 DOI: https://doi.org/10.1177/1098214010389321
Coryn, C. L. S., Ozeki, S., Wilson, L. N., Greenman II, G. D., Schröter, D. C., Hobson, K. A., Azzam, T., & Vo, A. T. (2016). Does research on evaluation matter? Findings from a survey of American Evaluation Association members and prominent evaluation theorists and scholars. American Journal of Evaluation, 37(2), 159-173. https://doi.org/10.1177/1098214015611245 DOI: https://doi.org/10.1177/1098214015611245
Cronbach, L. J. (1963). Course improvement through evaluation. Teachers College Record, 64, 672-683. https://doi.org/10.1177/016146816306400802 DOI: https://doi.org/10.1177/016146816306400802
Cronbach, L. J. (1982). Designing evaluations of educational and social programs. San Francisco: Jossey-Bass.
Davidson, E. J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage. https://doi.org/10.4135/9781452230115 DOI: https://doi.org/10.4135/9781452230115
Evers, J. W. (1980). A field study of goal-based and goal-free evaluation techniques. Unpublished doctoral dissertation, Western Michigan University, Kalamazoo.
Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2004). Program evaluation: Alternative approaches and practical guidelines (3rd ed.). Boston: Pearson Education.
Fournier, D. M. (Ed.). (1995). Establishing evaluative conclusions: A distinction between general and working logic. New Directions for Evaluation, 68, 15-32. San Francisco: Jossey-Boss. https://doi.org/10.1002/ev.1017 DOI: https://doi.org/10.1002/ev.1017
Fournier, D. M. (2005). Logic of evaluation: Working logic. In S. Mathison (Ed.), Encyclopedia of evaluation (pp. 238-242). Thousand Oaks, CA: Sage.
Friedman, V. J., Rothman, J. & Withers, B. (2006). The power of why: Engaging the goal paradox in program evaluation. American Journal of Evaluation, (27)2, 201-218. https://doi.org/10.1177/1098214006288284 DOI: https://doi.org/10.1177/1098214006288284
Grinnell, R. M., Unrau, Y.A., & Gabor, P. A. (2011). Program evaluation. In R. Grinnell & Y. Unrau (Eds.) Social work research and evaluation: Foundations of evidence-based practice, (9th ed). (pp.521-529). New York: Oxford University Press.
Hellström, T. & Jacob, M. (2003). Knowledge without goals? Evaluation of knowledge management programmes. Evaluation, 9(1), 55-72. https://doi.org/10.1177/1356389003009001004 DOI: https://doi.org/10.1177/1356389003009001004
Himmelfarb, S. (1993). The measurement of attitudes. In A.H. Eagly & S. Chaiken (Eds.), Psychology of Attitudes, (pp. 23-88). Thomson/Wadsworth.
House, E. R. (1980). Evaluating with validity. Beverly Hills, CA: Sage.
House, E. R. (1983). Assumptions underlying evaluation models. In G.F. Madaus, M. Scriven, & D. L. Stufflebeam (Eds.), Evaluation Models. Boston: Kluwer-Nijhoff. https://doi.org/10.1007/978-94-009-6675-8_3 DOI: https://doi.org/10.1007/978-94-009-6675-8_3
Irving, J. F. (1979). Goal-free evaluation: Philosophical and ethical aspects of Michael Scriven's Model. CEDR quarterly, 12(3), 11-14.
James, C. & Roffe, I. (2000). The evaluation of goal and goal-free training innovation. Journal of European Industrial Training, 24(1), 12-20. https://doi.org/10.1108/03090590010308228 DOI: https://doi.org/10.1108/03090590010308228
Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards: How to assess evaluations of educational programs (2nd ed.). Thousand Oaks, CA: Sage.
Julnes, G. (2012). Managing valuation. In G. Julnes (Ed.), Promoting valuation in the public interest: Informing policies for judging value in evaluation. New Directions for Evaluation, 133, 3-15. https://doi.org/10.1002/ev.20002 DOI: https://doi.org/10.1002/ev.20002
Kundin, D. M. (2010). A conceptual framework for how evaluators make everyday practice decisions. American Journal of Evaluation, 31(3), 347-362. https://doi.org/10.1177/1098214010366048 DOI: https://doi.org/10.1177/1098214010366048
Madaus, G. F., & Stufflebeam, D. L. (1989). Educational evaluation: Classic works of Ralph W. Tyler. Boston: Kluwer Academic Publishers. https://doi.org/10.1007/978-94-009-2679-0 DOI: https://doi.org/10.1007/978-94-009-2679-0
Manfredi, T. C. (2003). Goal based or goal free evaluation? Growing New Farmers website. http://www.smallfarm.org/uploads/uploads/Files/Goal_Based_or_Goal_Free_Evaluation.pdf
Mark, M. M., Henry, G. T., & Julnes, G. (2000). Evaluation: An integrated framework for understanding, guiding, and improving policies and programs. San Francisco: Jossey-Bass.
Matsunaga, D. S., & Enos, R. (1997). Goal-free evaluation of the Alger Foundation's Wai'anae Self-help Housing Project: Evaluation report. Kalamazoo, MI: Western Michigan University, The Evaluation Center.
Metfessel, N. S., & Michael, W.B. (1967). A paradigm involving multiple criterion measures for the evaluation of the effectiveness of school programs. Educational and Psychological Measurement, 27, 931-943. https://doi.org/10.1177/001316446702700441 DOI: https://doi.org/10.1177/001316446702700441
Mowbray, C. T, Holter, M.C., Teague, G. B. & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24(3), 315-340. https://doi.org/10.1177/109821400302400303 DOI: https://doi.org/10.1177/109821400302400303
Osgood, C. E., Suci, G., & Tannenbaum, P. (1957). The measurement of meaning. Urbana, IL: University of Illinois Press.
Patton, M. Q. (1988). The evaluator's responsibility for utilization. Evaluation Practice, 9(2), 5-24. https://doi.org/10.1016/S0886-1633(88)80059-X DOI: https://doi.org/10.1016/S0886-1633(88)80059-X
Patton, M. Q. (1997). Utilization-focused evaluation: The new century text (3rd ed.). Thousand Oaks, CA: Sage Publications.
Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.
Popham, W. J., Eisner, E. W., Sullivan, H. J., & Tyler, L. L. (1969). Instructional objectives (American Educational Research Association Monograph Series on Curriculum Evaluation no. 3). Chicago: Rand McNally.
Powell, E. R. (1982). Sociometric semantic differential assessment. Small Group Research, 13(1), 43-52. https://doi.org/10.1177/104649648201300103 DOI: https://doi.org/10.1177/104649648201300103
Scriven, M. (1967). The methodology of evaluation. In R. E. Stake (Ed.), Curriculum evaluation. American Educational Research Association Monograph Series on Evaluation No. 1. Chicago: Rand McNally.
Scriven, M. (1972). Pros and cons about goal-free evaluation. The Journal of Educational Evaluation, 3(4), 1-7.
Scriven, M. (1973). Goal-free evaluation. In E. R. House (Ed.), School evaluation: The politics and process (pp. 319-328). Berkeley, CA: McCutchan Publishing Corporation.
Scriven, M. (1974). Prose and cons about goal-free evaluation. In W.J. Popham (Ed.), Evaluation in education: Current applications (pp. 34-67). Berkeley, CA: McCutchan Publishing Corporation.
Scriven, M. (1976). Evaluation bias and its control. In G. V. Glass (Ed.), Evaluation studies review annual (Vol. 1) (pp. 101-118). Newbury Park, CA: Sage.
Scriven, M. (1991). Evaluation thesaurus (4th ed.). Newbury Park, CA: Sage Publication.
Scriven, M. (2005). The problem of free will in program evaluation. Journal of MultiDisciplinary Evaluation, 2(2), 102-104. https://doi.org/10.56645/jmde.v2i2.124 DOI: https://doi.org/10.56645/jmde.v2i2.124
Shadish, W. R., Cook, T. D., & Leviton, L. C. (1991). Foundations of program evaluation. Newbury Park, CA: Sage.
Shulha, L. M. & Cousins, J. B. (1997). Evaluation use: Theory, research and practice since 1986. Evaluation Practice, 18(3), 195-208. https://doi.org/10.1016/S0886-1633(97)90027-1 DOI: https://doi.org/10.1016/S0886-1633(97)90027-1
Stake, R. E. (1967). The countenance of educational evaluation. Teacher College Record, 68, 523-540. https://doi.org/10.1177/016146816706800707 DOI: https://doi.org/10.1177/016146816706800707
Stufflebeam, D. L. (2001). The metaevaluation imperative. American Journal of Evaluation, 20(2), 183-209. https://doi.org/10.1016/S1098-2140(01)00127-8 DOI: https://doi.org/10.1016/S1098-2140(01)00127-8
Stufflebeam, D. L., & Coryn, C. L. S. (2014). Evaluation theory, models, & applications (2nd ed.). San Francisco, CA: Jossey-Bass.
Stufflebeam, D. L. & Shinkfield, A. J. (2007). Evaluation theory, models, and applications. San Francisco: Jossey-Bass.
Suchman, E. (1967). Evaluative research: Principles and practice in public service and social action programs. New York: Russell Sage.
Suchman, E. (1969). Evaluating educational programs. Urban Review 3(4), 15-17. https://doi.org/10.1007/BF02322249 DOI: https://doi.org/10.1007/BF02322249
Thiagarajan, S. (1975). Goal-free evaluation of media. Educational Technology, (15)5, 38-40.
Tourmen, C. (2009). Evaluators' decision making: The relationship between theory, practice, and experience. American Journal of Evaluation, 30(1), 7-30. https://doi.org/10.1177/1098214008327602 DOI: https://doi.org/10.1177/1098214008327602
Vedung, E. (1997). Public policy and program evaluation. New Brunswick, NJ: Transaction Publishers.
Welch, W. W. (1976). ''Goal Free Evaluation Report for St. Mary's Junior College'' (Unpublished Report) Minneapolis.
Welch, W. W. (1978, March). Goal-free formative evaluation - an example. Paper presented at the meeting of the American Education Research Association, Toronto, Ontario, Canada.
Weiss, C. H. (1998). Have we learned anything new about the use of evaluation? American Journal of Evaluation, 19(1), 21-33. https://doi.org/10.1016/S1098-2140(99)80178-7 DOI: https://doi.org/10.1016/S1098-2140(99)80178-7
Worthen, B. (1990). Program evaluation. H. Walberg & G. Haertel (Eds.), The international encyclopedia of educational evaluation (pp. 42-47). Toronto, ON: Pergammon Press.
Youker, B. W. (2005). Goal-free evaluation of the 2005 Kalamazoo Public School Middle School Summer Enrichment Program: Final report. Kalamazoo, MI: Western Michigan University, The Evaluation Center.
Youker, B. W. (2011). An analog experiment comparing goal-free evaluation and goal achievement evaluation utility (Unpublished doctoral dissertation). Western Michigan University, Kalamazoo.
Youker, B. W. (2013). Goal-free evaluation: A potential model for the evaluation of social work programs. Social Work Research, 37(4), 432-438. https://doi.org/10.1093/swr/svt032 DOI: https://doi.org/10.1093/swr/svt032
Youker, B. W. & Ingraham, A. (2013). Goal-free evaluation: An orientation for foundations' evaluations. The Foundation Review, 5(4), 53-63. https://doi.org/10.9707/1944-5660.1182 DOI: https://doi.org/10.9707/1944-5660.1182
Youker, B. W., Ingraham, A., & Bayer, N. (2014). An assessment of goal-free evaluation: Case studies of four goal-free evaluations. Evaluation and Program Planning, 46, 10-16. https://doi.org/10.1016/j.evalprogplan.2014.05.002 DOI: https://doi.org/10.1016/j.evalprogplan.2014.05.002