How Good are Our Measures? Investigating the Appropriate Use of Factor Analysis for Survey Instruments
Main Article Content
Abstract
Background: Evaluation work frequently utilizes factor analysis to establish the dimensionality, reliability, and stability of surveys. However, survey data is typically ordinal, violating the assumptions of most statistical methods, and thus is often factor-analyzed inappropriately.
Purpose: This study illustrates the salient analytical decisions for factor-analyzing ordinal survey data appropriately and demonstrates the repercussions of inappropriate analyses.
Setting: The data used for this study are drawn from an evaluation of the efficacy of a drama-based approach to teaching Shakespeare in elementary and middle school.
Intervention: Not applicable.
Research Design: Survey research.
Data Collection and Analysis: Four factor analytic methods were compared: a traditional exploratory factor analysis (EFA), a full-information EFA, and two EFAs within the confirmatory factor analysis framework (E/CFA) conducted according to the Jöreskog method and the Gugiu method.
Findings: Methods appropriate for ordinal data produce better models, the E/CFAs outperform the EFAs, and the Gugiu method demonstrates greater model interpretability and stability than the Jöreskog method. These results suggest that the Gugiu E/CFA may be the preferable factor analytic method for use with ordinal data. Practical applications of these findings are discussed.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Brown, T. A. (2006). Confirmatory factor analysis for applied research. Guilford Press.
Brown, T. A., White, K. S., Forsyth, J. P., & Barlow, D. H. (2005). The structure of perceived emotional control: Psychometric properties of a revised Anxiety Control Questionnaire. Behavior Therapy, 35(1), 75-99. DOI: https://doi.org/10.1016/S0005-7894(04)80005-4
https://doi.org/10.1016/S0005-7894(04)80005-4 DOI: https://doi.org/10.1016/S0005-7894(04)80005-4
Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136-162). Beverly Hills, CA: Sage.
DiStefano, C. (2002). The impact of categorization with confirmatory factor analysis. Structural Equation Modeling, 9(3), 327-346. DOI: https://doi.org/10.1207/S15328007SEM0903_2
https://doi.org/10.1207/S15328007SEM0903_2 DOI: https://doi.org/10.1207/S15328007SEM0903_2
Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272-299. DOI: https://doi.org/10.1037/1082-989X.4.3.272
https://doi.org/10.1037//1082-989X.4.3.272 DOI: https://doi.org/10.1037//1082-989X.4.3.272
Gilley, W., & Uhlig, G. (1993). Factor analysis and ordinal data. Education, 114(2), 258-264.
Glorfeld, L. W. (1995). An improvement on Horn's parallel analysis methodology for selecting the correct number of factors to retain. Educational and Psychological Measurement, 55, 377-393. DOI: https://doi.org/10.1177/0013164495055003002
https://doi.org/10.1177/0013164495055003002 DOI: https://doi.org/10.1177/0013164495055003002
Gugiu, P. C. (2011). Exploratory Factor Analysis within a Confirmatory Factor Analysis Framework (E/CFA). Expert lecture presented at the 2011 American Evaluation Association conference in Anaheim, California.
Gugiu, P. C., Coryn, C., Clark, R., & Kuehn, A. (2009). Development and evaluation of the short version of the Patient Assessment of Chronic Illness Care instrument. Chronic Illness, 5(4), 268-276. DOI: https://doi.org/10.1177/1742395309348072
https://doi.org/10.1177/1742395309348072 DOI: https://doi.org/10.1177/1742395309348072
Hayton, J. C., Allen, D. G., & Scarpello, V. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7(2), 191-205. DOI: https://doi.org/10.1177/1094428104263675
https://doi.org/10.1177/1094428104263675 DOI: https://doi.org/10.1177/1094428104263675
Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 32, 179-185. DOI: https://doi.org/10.1007/BF02289447
https://doi.org/10.1007/BF02289447 DOI: https://doi.org/10.1007/BF02289447
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. DOI: https://doi.org/10.1080/10705519909540118
https://doi.org/10.1080/10705519909540118 DOI: https://doi.org/10.1080/10705519909540118
Jöreskog, K. G. (1969). A general approach to confirmatory maximum likelihood factor analysis. Psychometrika, 34(2), 183-202. DOI: https://doi.org/10.1007/BF02289343
https://doi.org/10.1007/BF02289343 DOI: https://doi.org/10.1007/BF02289343
Jöreskog, K. G. & Moustaki, I. (2006). Factor analysis of ordinal variables with full information maximum likelihood. Hillsdale, NJ: Lawrence Erlbaum Associates.
Jöreskog, K. G., & Sörbom, D. (1979). Advances in factor analysis and structural equation models (p. 105). J. Magidson (Ed.). Cambridge, MA: Abt Books.
Jöreskog, K. G., & Sörbom, D. (1996). LISREL 8: User's reference guide. Lincolnwood, IL: Scientific Software International.
Kampen, J., & Swyngedouw, M. (2000). The ordinal controversy revisited. Quality and Quantity, 34(1), 87-102. DOI: https://doi.org/10.1023/A:1004785723554
https://doi.org/10.1023/A:1004785723554 DOI: https://doi.org/10.1023/A:1004785723554
Lee, B. K., Patall, E. A., Cawthon, S. W., & Steingut, R. R. (2015). The effect of drama-based pedagogy on preK-16 outcomes: A meta-analysis of research from 1985 to 2012. Review of Educational Research, 85, 3-49. DOI: https://doi.org/10.3102/0034654314540477
https://doi.org/10.3102/0034654314540477 DOI: https://doi.org/10.3102/0034654314540477
Olsson U. (1979a). Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika, 44, 443-460. DOI: https://doi.org/10.1007/BF02296207
https://doi.org/10.1007/BF02296207 DOI: https://doi.org/10.1007/BF02296207
Olsson, U. (1979b). On the robustness of factor analysis against crude classification of the observations. Multivariate Behavioral Research, 14(4), 485-500. DOI: https://doi.org/10.1207/s15327906mbr1404_7
https://doi.org/10.1207/s15327906mbr1404_7 DOI: https://doi.org/10.1207/s15327906mbr1404_7
Raykov, T. (2001). Estimation of congeneric scale reliability using covariance structure analysis with nonlinear constraints. British Journal of Mathematical and Statistical Psychology, 54(2), 315-323. DOI: https://doi.org/10.1348/000711001159582
https://doi.org/10.1348/000711001159582 DOI: https://doi.org/10.1348/000711001159582
Raykov, T. (2004). Behavioral scale reliability and measurement invariance evaluation using latent variable modeling. Behavior Therapy, 35(2), 299-331. DOI: https://doi.org/10.1016/S0005-7894(04)80041-8
https://doi.org/10.1016/S0005-7894(04)80041-8 DOI: https://doi.org/10.1016/S0005-7894(04)80041-8
Satorra, A., & Bentler, P. M. (1994). Corrections to test statistics and standard errors in covariance structure analysis. In A. von Eye & C. Clogg (Eds.), Latent variable analysis: Applications for developmental research (pp. 399-419). Thousand Oaks, CA: Sage.
Schumacker, R. E. & Lomax, R. G. (2010). A beginner's guide to SEM. Manwah, NJ: Lawrence Erlbaum Associates.
Steiger, J. H., & Lind, J. C. (1980, May). Statistically-based tests for the number of common factors. Paper presented at the annual meeting of the Psychometric Society, Iowa City, IA.
Strand, S. (2009). Attitude to Shakespeare among Y10 students: Final report to the Royal Shakespeare Company on the Learning and Performance Network student survey 2007-2009. Warwick, England: Centre for Educational Development, Appraisal and Research.
Tabachnick, B. G., & Fidell, L. S. (2001). Using multivariate statistics. Boston: Pearson.
Tucker, L. R., & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1-10. DOI: https://doi.org/10.1007/BF02291170
https://doi.org/10.1007/BF02291170 DOI: https://doi.org/10.1007/BF02291170
Yeomans-Maldonado, G., Gugiu, C. P., & Enciso, P. (2013). Using the Rasch model to assess the interest for Shakespeare and illustrate rating scale diagnostics. Poster presented at the Modern Modeling Methods Conference, Storrs, CT.
Yuan, Y. C. (2000). Multiple imputation for missing data: Concepts and new development (SAS Tech. Rep. No. P267-25). Rockville, MD: SAS Institute.
Zumbo, B. D., Gadermann, A. M., & Zeisser, C. (2007). Ordinal versions of coefficients alpha and theta for Likert rating scales. Journal of Modern Applied Statistical Methods, 6, 21-29. DOI: https://doi.org/10.22237/jmasm/1177992180
https://doi.org/10.22237/jmasm/1177992180 DOI: https://doi.org/10.22237/jmasm/1177992180