Benefits of the Person-Oriented Perspective for Program Evaluation: Analyzing the Differential Treatment Effects of the Vienna E-Lecturing Program
Main Article Content
Abstract
Background: In program evaluation interventions are usually examined for global effects but not always for differential effects. The reasoning behind the focus on global effects is that most of the concepts applied in data analyses, and most of the hypotheses being assessed, are variable-oriented. Consequently, existing differential effects will not be detected through global analyses based on a variable-oriented perspective. Accordingly, the intervention under evaluation will be wrongly considered to be ineffective.
Purpose: This contribution, therefore, argues for the application of person-oriented analyses in addition to the common variable-oriented approach in order to identify the differential effects of an intervention during evaluation.
Setting: Both approaches will be applied in the evaluation of a showcase program conducted in real life setting. The results of the two perspectives will be compared.
Intervention: An internet supported program for the promotion of self-regulated learning in an academic context called Vienna E-Lecturing (VEL).
Research Design: The evaluation of the VEL was conducted with a treatment-control-design. Data was collected at the beginning as well as at the end of the intervention.
Data Collection and Analysis: Data regarding motivation as well as factual knowledge was collected. All motivational variables were assessed using previously published scales consisting of 3-4 items. Factual knowledge in statistics and methodology was tested regarding the aspects of reproduction, comprehension and the production of methodological content.
Findings: The application of a variable-oriented perspective showed small effects with low practical relevance. Through the application of a person-oriented approach, and taking into account the theoretical framework of the intervention, three students groups which differed systematically in their learning motivation could be identified. For two of these groups the intervention was very successful. The discussion provides recommendations on how person-oriented approaches can be used systematically in program evaluation.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
Funding data
-
Oesterreichische Nationalbank
Grant numbers 11107
References
Bergman, L. R., & Magnusson, D., (19970. A person-oriented approach in research on developmental psychopathology. Development and Psychopathology, 9, 291-319. https://doi.org/10.1017/S095457949700206X DOI: https://doi.org/10.1017/S095457949700206X
Bergman, L. R., Magnusson, D., & El- Khouri, B. M., (2003). Studying individual development in an interindividual context. A person-oriented approach. Mahwah, NJ: Lawrence Erlbaum Associates. https://doi.org/10.4324/9781410606822 DOI: https://doi.org/10.4324/9781410606822
Blalock, H. M. Jr., (1987). Some general goals in teaching statistics. Teaching Sociology, 15, 164-172. https://doi.org/10.2307/1318031 DOI: https://doi.org/10.2307/1318031
Bloom, B. S., & Krathwohl, D. R., (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook 1: Cognitive domain. New York , Longmans.
Bogat, G. A., Levendosky, A. A., & von Eye, A., (2005). The future of research on intimate partner violence - person- oriented and variable-oriented perspectives. American Journal of Community Psychology. 36, 49-70. https://doi.org/10.1007/s10464-005-6232-7 DOI: https://doi.org/10.1007/s10464-005-6232-7
Breitkopf, L. (1985). Die Hilflosigkeitsskala. Diagnostica, 31, 221-233.
Chen, H. (1990). Theory-driven evaluations. Newbury Park, CA: Sage.
Chen, H. (2006). A theory-driven evaluation perspective on mixed methods research. Research in the Schools, 13, 75-83.
Dweck, C. S., Chiu, C., & Hong, Y. (1995). Implicit theories and their role in judgements and reactions: A world from two perspectives. Psychological Inquiry, 6, 267-285. https://doi.org/10.1207/s15327965pli0604_1 DOI: https://doi.org/10.1207/s15327965pli0604_1
Dweck, C. S., & Henderson, V. L., (1988). Theories of intelligence: Background and measures. Unpublished manuscript, University of Illinois.
Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation and personality. Psychological Review, 95, 256-273. https://doi.org/10.1037/0033-295X.95.2.256 DOI: https://doi.org/10.1037//0033-295X.95.2.256
Fagan, A., & Mihalic, S. (2003). Strategies for enhancing the adoption of school- based prevention programs - lessons learned from the blueprints for violence prevention replications of the life skills training program. Journal of Community Psychology, 31, 235-253. https://doi.org/10.1002/jcop.10045 DOI: https://doi.org/10.1002/jcop.10045
Gal, I., & Ginsburg, L. (1994). The role of beliefs and attitudes in learning statistics: Towards an assessment framework. Journal of Statistics Education, 2, 1-54. https://doi.org/10.1080/10691898.1994.11910471 DOI: https://doi.org/10.1080/10691898.1994.11910471
Gibson, C. M. (2003). Privileging the participant: The importance of sub- group analysis in social welfare evaluations. American Journal of Evaluation, 24, 443-469. https://doi.org/10.1177/109821400302400403 DOI: https://doi.org/10.1016/j.ameval.2003.09.002
Greene, J., Benjamin, L., & Goodyear, L. (2001). The merits of mixing methods in evaluation. Evaluation, 7, 25-44. https://doi.org/10.1177/13563890122209504 DOI: https://doi.org/10.1177/13563890122209504
Jerusalem, M., & Satow, L. (1999). Schulbezogene Selbstwirksamkeitserwartung (WIRKSCHUL). In R. Schwarzer & M. Jerusalem (Ed.), Skalen zur Erfassung von Lehrer- und Schülermerkmalen. Berlin: FU.
Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards (2nd ed.). Thousand Oaks, CA: Sage.
Lapka, D., Wagner, P., Schober, B., Gradinger, P., & Spiel, C. (2010). Methodenlehre: Alptraum oder Herausforderung für Psychologiestudierende? Eine Typologie auf Basis des sozialkognitiven Motivationsmodells von Dweck, [Methodology classes: Nightmare or a challenge for Psychology students? A typology based on Dweck's social-cognitive approach to motivation]. Psychologie in Erziehung und Unterricht, 57(3). https://doi.org/10.2378/peu2010.art15d DOI: https://doi.org/10.2378/peu2010.art15d
Lienert,G.A.,&Krauth,J.(1973).KFA. Die Konfigurationsfrequenzanalyse und ihre Anwendung in Psychologie und Medizin, [The Configural Frequency Analysis and its application in Psychology and Medicine]. Freiburg: Alber.
Lucke, J., Donald, M., Dower, J., & Raphael, B. (2001). Considerations in the design of a mixed-method cluster. Evaluation, 7, 110-131. https://doi.org/10.1177/13563890122209559 DOI: https://doi.org/10.1177/13563890122209559
Magnusson, D. (2000). The individual as the organizing principle in psychological inquiry: A holistic approach. In: L.R. Bergman, R.B. Cairns, L.G. Nilsson & L. Nystedt (eds.), Developmental science and the holistic approach. Mahwah, NJ: Erlbaum, pp.33-47.
Morgan, P. L., Farkas, G., & Hibel, J. (2008). Matthew effects for whom? Learning Disability Quarterly, 31, 187-198 https://doi.org/10.2307/25474651 DOI: https://doi.org/10.2307/25474651
Onwuegbuzie, A. J. (2000). Statistics anxiety and the role of self- perceptions. The Journal of Educational Research, 93, 323-330. https://doi.org/10.1080/00220670009598724 DOI: https://doi.org/10.1080/00220670009598724
Patton, M.Q. (1996). Utilization-focused Evaluation (3rd ed.). Thousand Oaks, CA: Sage.
Peck, L. R. (2005). Using cluster analysis in program evaluation. Evaluation Review, 29, 178-196 https://doi.org/10.1177/0193841X04266335 DOI: https://doi.org/10.1177/0193841X04266335
Schiefele, U., Krapp, A., Wild, K.-P., & Winteler, A. (1993). Der "Fragebogen zum Studieninteresse"(FSI). Diagnostica, 39(4), 335-351. https://doi.org/10.1037/t64341-000 DOI: https://doi.org/10.1037/t64341-000
Schober, B. (2002). Entwicklung und Evaluation des Münchner Motivationstrainings (MMT) (Theorie und Forschung Bd. 763). Regensburg: Roderer.
Schober, B., Wagner, P., Reimann, R., Atria, M., & Spiel, C. (2006). Teaching research methods in an internet-based blended-learning setting. Methodology, 2, 73-82. https://doi.org/10.1027/1614-2241.2.2.73 DOI: https://doi.org/10.1027/1614-2241.2.2.73
Schmidt, M., Perels, F., & Schmitz, B. (2010). How to perform idiographic and a combination of idiographic and nomothetic approaches. Journal of Psychology, 218, 166-174. https://doi.org/10.1027/0044-3409/a000026 DOI: https://doi.org/10.1027/0044-3409/a000026
Spiel, C. (1998). Four methodological approaches for the study of stability and change in development. Methods of Psychological Research - online 3 (2), http://www.ppm.ipn.uni- kiel.de/mpr/issue5/art2/article.html (acessed November 12, 2001).
Spiel, C., Strohmeier, D., & Atria, M. (2008). Von der notwendigen Verschränkung quantitativer und qualitativer Methoden in der Programmevaluation am Beispiel des WiSK - Soziales Kompetenztraining für Schülerinnen und Schüler. In: F. Hofmann, C. Schreiner and J. Thonhauser eds. 2008. Zur Komplementarität von qualitativen und quantitativen Aspekten in der erziehungswissenschaftlichen Forschung, [On the need to offset quantitative and qualitative methods in program evaluation using WiSK as an example - Social competence training for male and female pupils]. Münster: Waxmann, pp.263-280.
Spiel, C., Lapka, D., Gradinger, P., Zodlhofer, E. M., Reimann, R., Schober, B., Wagner, P., & von Eye, A. (20080. A euclidean distance-based matching procedure for non- randomized comparison studies. European Psychologist, 13(3), 180- 187. https://doi.org/10.1027/1016-9040.13.3.180 DOI: https://doi.org/10.1027/1016-9040.13.3.180
Sterba, S. K., & Bauer, D. J. (2010). Matching method with theor in person-oriented developmental psychopathology research. Development and Psychopathology, 22, 239-254. https://doi.org/10.1017/S0954579410000015 DOI: https://doi.org/10.1017/S0954579410000015
von Eye, A., & Bogat, G. A. (2006). Person-oriented and variable-oriented research: Concepts, results, and development. Merrill-Palmer Quarterly, 52, 390-420. https://doi.org/10.1353/mpq.2006.0032 DOI: https://doi.org/10.1353/mpq.2006.0032
von Eye, A., & Bergman, L. R. (2003). Research strategies in developmental psychopathology: Dimensional identity and the person-oriented approach. Development and Psychopathology, 15, 553-580. https://doi.org/10.1017/S0954579403000294 DOI: https://doi.org/10.1017/S0954579403000294
von Eye, A., Bogat, G. A., & Rhodes, J. E. (2006). Variable-oriented and person- oriented perspectives of analysis: The example of alcohol consumption in adolescence. Journal of Adolescence, 29, 981-1004. https://doi.org/10.1016/j.adolescence.2006.06.007 DOI: https://doi.org/10.1016/j.adolescence.2006.06.007
von Eye, A., & Spiel, C. (2010). Conducting person-oriented research. Journal of Psychology, 218, 151-154. https://doi.org/10.1027/0044-3409/a000024 DOI: https://doi.org/10.1027/0044-3409/a000024
von Eye, A., Spiel, C., & Wood, P.K. (1996). CFA models, tests, interpretation, and alternatives-a rejoinder. Applied Psychology, 45, 345-352. https://doi.org/10.1111/j.1464-0597.1996.tb00776.x DOI: https://doi.org/10.1111/j.1464-0597.1996.tb00776.x