Using Cognitive Interviewing to Test Youth Survey and Interview Items in Evaluation: A Case Example
Main Article Content
Abstract
Background: Cognitive interviewing is a pretesting tool used by evaluators to increase item and response option validity. Cognitive interviewing techniques are used to assess the cognitive processes utilized by participants to respond to items. This approach is particularly appropriate for testing items with children and adolescents who have more limited cognitive capacities than adults, vary in their cognitive development, and have a unique perspective on their life experiences and context.
Purpose: This paper presents a case example of cognitive interviewing with youth as part of a national program evaluation, and aims to expand the use of cognitive interviewing as a pretesting tool for both quantitative and qualitative items in evaluation studies involving youth.
Setting: Youth participants were located in four regions of the United States: Northeast, Central, Southern, and Western. Interviewers were located at Montclair State University.
Intervention: Not applicable.
Research design: A cognitive interview measure was designed to include a subset of survey items, interview questions, and verbal probes, to evaluate if these items and questions would be understood as intended by both younger and older youth participants. An iterative design was used with cognitive interviewing testing rounds, analysis, and revisions.
Data Collection and Analysis: The cognitive interview was administered by phone to 10 male youth, five from the 10-13-year-old age range and five from the 15-17-year-old age range. Interviews were audio-recorded, transcribed, reviewed, and coded. Survey items and interview questions were revised based on feedback from the participants and consensus agreement among the evaluation team. Item revisions were included in further testing rounds with new participants.
Findings: As a result of using cognitive interviewing to pretest survey and interview items with youth, response errors were identified. Participants did not understand some of the items and response options as intended, indicating problems with validity. These findings support the use of cognitive interviewing for testing and modifying survey items adapted for use with youth, as well as qualitative interview items. Additionally, the perspective of the youth participants was valuable for informing decisions to modify items and helping the evaluators learn the participants’ program culture and experiences. Based on the findings and limitations of the study, we give practice recommendations for future studies using cognitive interviewing with a youth sample.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
Funding data
-
S. D. Bechtel, Jr. Foundation and Stephen Bechtel Fund
Grant numbers 8325
References
Beatty, P.C., & Willis, G.B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 7(2), 287-311. https://doi.org/10.1093/poq/nfm006 DOI: https://doi.org/10.1093/poq/nfm006
Blair, J. & Conrad, F. G. (2011). Sample size for cognitive interview pretesting. Public Opinion Quarterly, 75(4), 636-658. https://doi.org/10.1093/poq/nfr035 DOI: https://doi.org/10.1093/poq/nfr035
Boeiji, H. & Willis, G. (2013). The Cognitive Interviewing Reporting Framework (CIRF): Towards the harmonization of cognitive testing reports. Methodology, 9(3), 87-95. https://doi.org/10.1027/1614-2241/a000075 DOI: https://doi.org/10.1027/1614-2241/a000075
Borgers, N., de Leeuw, E., & Hox, J., (2000). Children as respondents in survey research: Cognitive development and response quality. Bulletin of Sociological Methodology, 66, 60-75. http://pascal-francis.inist.fr/vibad/index.php?action=search&terms=1551123 https://doi.org/10.1177/075910630006600106 DOI: https://doi.org/10.1177/075910630006600106
Bos, N., Engel, M. F., van Rijswijk, N. J., Verheijden, J. M. A., Coster, W., Moed, R., & Ketelaar, M. (2019). Translation and cross-cultural adaptation of the PEDI-CAT: Dutch version. Journal of Pediatric Rehabilitation Medicine: An Interdisciplinary Approach, 12, 57-64. doi: 10.3233/PRM-180544 DOI: https://doi.org/10.3233/PRM-180544
Buers, C., Triemstra, M., Bloemendal, E., Zwijnenberg, N. C., Hendriks, M., & Delnoij, D. M. (2014). The value of cognitive interviewing for optimizing a patient experience survey. International Journal of Social Research Methodology, 17(4), 325-340. https://doi.org/10.1080/13645579.2012.750830 DOI: https://doi.org/10.1080/13645579.2012.750830
Campanelli, P., Gray, M., Blake, M., & Hope, S. (2015). Cognitive interviewing as tool for enhancing the accuracy of the interpretation of quantitative findings. Quality & Quantity, 50(3). 1021-1040. https://doi.org/10.1007/s11135-015-0188-y DOI: https://doi.org/10.1007/s11135-015-0188-y
Castillo-Díaz, M., & Padilla, J. (2013). How cognitive interviewing can provide validity evidence of the response processes to scale items. Social Indicators Research, 114(3), 963-975. https://doi.org/10.1007/s11205-012-0184-8 DOI: https://doi.org/10.1007/s11205-012-0184-8
DeCastellarnau, A. (2018). A classification of response scale characteristics that affect data quality: A literature review. Quality & Quantity, 52, 1523–1559. https://doi.org/10.1007/s11135-017-0533-4 DOI: https://doi.org/10.1007/s11135-017-0533-4
De Leeuw, E., Borgers, N., & Smits, A. (2004). Pretesting questionnaires for children and adolescents. In Presser, S., Rothgeb, J. M., Couper, M. P., Lessler, J. T., Martin, E., Martin, J., & Singer, E. (Eds). Methods for testing and evaluating survey questionnaires. (pp. 409-430). John Wiley & Sons, Inc. https://doi.org/10.1002/0471654728.ch20 DOI: https://doi.org/10.1002/0471654728.ch20
Domanska, O. M., Firnges, C., Bollweg, T. M., SØresnsen, K., Holmberg, C., & Jordan, S. (2018). Do adolescents understand the items of the European Health Literacy Survey Questionnaire (HLS-EU-Q47) – German version? Findings from cognitive interviews of the project "Measurement of Health Literacy Among Adolescents" (MOHLAA) in Germany. Archives of Public Health, 76(46). https://doi.org/10.1186/s13690-018-0276-2 DOI: https://doi.org/10.1186/s13690-018-0276-2
Elkind, D. (2001). Cognitive Development. In J. V. Lerner & R. M. Lerner (Eds.), Adolescence in America: An encyclopedia (Vol. 1, pp. 127-134). ABC-CLIO, Inc.
Fowler, F. J. Jr., Lloyd, S. J., Cosenza, C. A., & Wilson, I. B. (2016). Coding cognitive interviews: An approach to enhancing the value of cognitive testing for survey question evaluation. Field Methods, 28(1), 3-20. https://doi.org/10.1177/1525822X14549921 DOI: https://doi.org/10.1177/1525822X14549921
Ginsburg, H. P., & Opper, S. (1988). Piaget's theory of intellectual development (3rd ed.). Prentice-Hall, Inc.
Hall, S., McKinstry, C., & Hyett, N. (2106). Youth perceptions of positive mental health. British Journal of Occupational Therapy, 79(8), 475-483. https://doi.org/10.1177/0308022616632775 DOI: https://doi.org/10.1177/0308022616632775
Hinds, J. T., III, Loukas, A., Chow, S., Pasch, K. E., Harrell, M. B., Perry, C. L., Delnevo, C., & Wackowski, O. A. (2016). Using cognitive interviewing to better assess young adult e-cigarette use. Nicotine & Tobacco Research, 18(10), 1998-2005. https://doi:10.1093/ntr/ntw096 DOI: https://doi.org/10.1093/ntr/ntw096
Hofmeyer, A., Sheingold, B. H., & Taylor, R. (2015). Do you understand what I mean? How cognitive interviewing can strengthen valid, reliable study instruments and dissemination products. Journal of International Education Research, 11(4), 261-267. https://doi.org/10.19030/jier.v11i4.9460 DOI: https://doi.org/10.19030/jier.v11i4.9460
Hsu, C. & Sandford, B. A. (2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research, and Evaluation, 12(10), 1-8. https://doi.org/10.7275/pdz9-th90
Irwin, C. W., & Stafford, E. T. (2016). Survey methods for educators: Collaborative survey development (part 1 of 3) (REL 2016–163). U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Northeast & Islands. http://ies.ed.gov/ncee/edlabs
Irwin, D. E., Varni, J. W., Yeatts, K., & DeWalt, D. A. (2009). Cognitive interviewing methodology in the development of a pediatric item bank: A patient reported outcomes measurement information system (PROMIS) study. Heath and Quality of Life Outcomes, 7(3), 1-10. https://doi.org/10.1186/1477-7525-7-3 DOI: https://doi.org/10.1186/1477-7525-7-3
Joffer, J., Jerdén, L., Öhman, A., & Flacking, R. (2016). Exploring self-rated health among adolescents: A think-aloud study. BMC Public Health, 16(156), 1-10. https://doi.org/10.1186/s12889-016-2837-z DOI: https://doi.org/10.1186/s12889-016-2837-z
Karabenick, S. A., Woolley, M. E., Friedel, J. M., Ammon, B. V., Blazevski, J., Bonney, C. R., De Groot, E., Gilbert, M. C, Musu, L., Kempler, T. M., & Kelly, K. L. (2007). Cognitive processing of self-report items in educational research: Do they think what we mean? Educational Psychologist, 42(3), 139-151. https://doi.org/10.1080/00461520701416231 DOI: https://doi.org/10.1080/00461520701416231
Karpen, S. C., & Hagemeier, N. E. (2017). Assessing faculty and student interpretations of AACP survey items with cognitive interviewing. American Journal of Pharmaceutical Education, 81(5), 1-9. https://doi.org/10.5688/ajpe81588 DOI: https://doi.org/10.5688/ajpe81588
Katz, C., McLeigh, J. D., & szwec, B. E. (2017). Preschoolers' perceptions of neighborhood environment, safety, and help-seeking. Children and Youth Services Review, 73, 197-204. http://dx.doi.org/10.1016/j.childyouth.2016.12.016 DOI: https://doi.org/10.1016/j.childyouth.2016.12.016
Kenny, U., Molcho, M., & Kelly, C. (2017). Young people's perspectives in developing a survey item on factors that influence body image. Young, 25(4S), 54S-72S. https://doi.org/10.1177/1103308816674136 DOI: https://doi.org/10.1177/1103308816674136
Koskey, K. L. K., Karabenick, S. A., Woolley, M. E., Bonney, C. R., & Dever, B. V. (2010). Cognitive validity of students' self-reports of classroom mastery goal structure: What students are thinking and why it matters. Contemporary Educational Psychology, 35, 254-263. https://doi.org/10.1016/j.cedpsych.2010.05.004 DOI: https://doi.org/10.1016/j.cedpsych.2010.05.004
Kramer, J. M., & Schwartz, A. (2017). Refining the Pediatric Evaluation of Disability Inventory – Patient-Reported Outcome (PEDI-PRO) item candidates: Interpretation of a self-reported outcome measure of functional performance by young people with neurodevelopmental disabilities. Developmental Medicine and Child Neurology, 59, 1083-1088. https://doi.org/10.1111/dmcn.13482 DOI: https://doi.org/10.1111/dmcn.13482
McCoy, H. (2014). Using cognitive interviewing to explore causes for racial differences on the MAYSI-2. Crime & Delinquency, 60(5), 647-666. https://doi.org/10.1177/0011128710388922 DOI: https://doi.org/10.1177/0011128710388922
McDavid, J. C., Huse, I., & Hawthorn, L. R. L. (2013). Program evaluation and performance measurement: An introduction to practice (2nd ed.). Sage Publications, Inc.
Moely, B. E., Mercer, S. H., Ilustre, V., Miron, D., & McFarland, M. (2002). Psychometric properties and correlates of the Civic Attitudes and Skills Questionnaire (CASQ): A measure of students' attitudes related to service-learning. Michigan Journal of Community Service Learning, 8(2), 15-26. http://hdl.handle.net/2027/spo.3239521.0008.202
Mulcahey, M. J., Calhoun, C., Riley, A., & Haley, S. M. (2009). Children's reports of activity and participation after sustaining a spinal cord injury: A cognitive interviewing study. Developmental Neurorehabilitation, 12(4), 191-200. https://doi.org/10.1080/17518420902998177 DOI: https://doi.org/10.1080/17518420902998177
Newcomer, K. E., & Triplett, T. (2015). Using surveys. In K. E. Newcomer, H. P. Hatry, & J. S. Wholey (Eds.), Handbook of practical program evaluation (4th ed., pp. 344-382). John Wiley & Sons, Inc. https://doi.org/10.1002/9781119171386.ch14 DOI: https://doi.org/10.1002/9781119171386.ch14
Norris, A. E., Torres-Thomas, S., & Williams, E. T. (2014). Adapting cognitive interviewing for early adolescent Hispanic girls and sensitive topics. Hispanic Health Care International,12(3), 111-119. https://doi.org/10.1891/1540-4153.12.3.111 DOI: https://doi.org/10.1891/1540-4153.12.3.111
Pepper, D., Hodgen, J., Lamesoo, K., Koiv, P., & Tolboom, J. (2018). Think aloud: Using cognitive interviewing to validate the PISA assessment of student self-efficacy in mathematics. International Journal of Research & Method in Education, 41(1), 3-16. https://doi.org/10.1080/1743727X.2016.1238891 DOI: https://doi.org/10.1080/1743727X.2016.1238891
Powell, R. R. (2006). Evaluation research: An overview. Library Trends, 55(1), 102-120. https://muse-jhu-edu.ezproxy.montclair.edu/article/202358 https://doi.org/10.1353/lib.2006.0050 DOI: https://doi.org/10.1353/lib.2006.0050
Presser, S., Couper, M. P., Lessler, J. T., Martin, E., Martin, J., Rothgeb, J. M., & Singer, E. (2004). Methods for testing and evaluating survey questions. Public Opinion Quarterly, 68(1), 109–130. https://doi.org/10.1093/poq/nfh008 DOI: https://doi.org/10.1093/poq/nfh008
Rebok, G., Riley, C. F., Starfield, B., Green, B., Robertson, J., & Tambor, E. (2001). Elementary school-aged children's reports of their health: A cognitive interviewing study. Quality of Life Research, 10(1), 59-70. https://www.jstor.org/stable/4037607 https://doi.org/10.1023/A:1016693417166 DOI: https://doi.org/10.1023/A:1016693417166
Reeve, B. B., Willis, G., Shariff-Marco, S. N., Breen, N., Williams, D. R., Gee, G. C., Alegria, M., Takeuchi, D. T., Stapleton, M., & Levin, K. Y. (2011). Comparing cognitive interviewing and psychometric methods to evaluate a racial/ethnic discrimination scale. Field Methods, 23(4), 397-419. https://doi.org/10.1177/1525822X11416564 DOI: https://doi.org/10.1177/1525822X11416564
Roberts, H. (2017). Listening to children and hearing them. In P. Christensen & A. James (Eds)., Research with children: Perspectives and practices (3rd ed., pp. 142-159). Routledge.
Rossi, P. H., Lipsey, M. W., & Henry, G. T. (2019). Evaluation: A systematic approach (8th ed). Sage Publications, Inc.
Rothgeb, J., Willis, G., & Forsyth, B. (2007). Questionnaire pretesting methods: Do different techniques and different organizations produce similar results? Bulletin of Sociological Methodology, 96, 5-31. http://pascal-francis.inist.fr/vibad/index.php?action=search&terms=19560247 https://doi.org/10.1177/075910630709600103 DOI: https://doi.org/10.1177/075910630709600103
Ryan, K., Gannon-Slater, N., & Culbertson, M. J. (2012). Improving survey methods with cognitive interviews in small- and medium-scale evaluations. American Journal of Evaluation, 33(3), 414-430. https://doi.org/10.1177/1098214012441499 DOI: https://doi.org/10.1177/1098214012441499
Sabo, K. (Ed.). (2003) Youth participatory evaluation: A field in the making. New Directions in Evaluation, 98, 1-12. https://doi.org/10.1002/ev.80 DOI: https://doi.org/10.1002/ev.81
Scott, J. (2008). Children as respondents: The challenge for quantitative methods. In P. Christensen & A. James, (Eds.). Research with children: Perspectives and practices (2nd ed., pp. 87-108). Routledge/Taylor & Francis Group.
Sebastian, R. A., Ramos, M. M., Stumbo, S., Mcgrath, J., & Fairbrother, G. (2014). Measuring youth health engagement: Development of the Youth Engagement with Health Services Survey. Journal of Adolescent Health, 55(3), 334-340. https://doi.org/10.1016/j.jadohealth.2014.02.008 DOI: https://doi.org/10.1016/j.jadohealth.2014.02.008
Silva, J. R., Fera, B., Sudula, S., Koetzle, D., & Schwalbe, C. (2019). Cognitive interviewing to improve questionnaires for justice-involved youth. International Journal of Offender Therapy and Comparative Criminology, 63(10), 1931-1951. https://doi.org/10.1177/0306624X19839597 DOI: https://doi.org/10.1177/0306624X19839597
Tatian, P. (2016). Performance measurement to evaluation. Urban Institute. http://www.urban.org/sites/default/files/alfresco/publication-pdfs/2000555-Performance-Measurement-to-Evaluation-March-2016-Update.pdf
Teal, R., Enga, Z., Diehl, S. J., Rohweder, C. L., Kim, M., Dave, G., Durr, A., Wynn, M., Roman Isler, M., Corbie-Smith, G., & Weiner, B. J. (2015). Applying cognitive interviewing to inform measurement of partnership readiness: A new approach to strengthening community–academic research. Progress in Community Health Partnerships: Research, Education, and Action, 9(4), 513-519. doi:10.1353/cpr.2015.0083 DOI: https://doi.org/10.1353/cpr.2015.0083
Tourangeau, R. (1984). Cognitive sciences and survey methods. In T. B. Jabine, M. L. Straf, J. N. Tanur, & R. Tourangeau, (Eds.). Cognitive aspects of survey methodology: Building a bridge between disciplines (pp. 73-100). National Academy Press. http://doi.org/10.17226/930 DOI: https://doi.org/10.17226/930
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge University Press. https://doi.org/10.1017/CBO9780511819322 DOI: https://doi.org/10.1017/CBO9780511819322
Urban, J. B., Linver, M. R., Moroney, D., Nichols, T., Hargraves, M., Roberts, E. D., Quinn, J., Brown, M., Gama, L., Gorsky, D., Doubledee, R., & Cox, M. (2019). Developing and testing a theory of change for Boy Scouts of America. Applied Developmental Science. https://doi.org/10.1080/10888691.2020.1844008 DOI: https://doi.org/10.1080/10888691.2020.1844008
Urban, J. B., & Trochim, W. (2009). The role of evaluation in research-practice integration: Working toward the "Golden Spike." American Journal of Evaluation, 30(4), 538-553. https://doi-org.ezproxy.montclair.edu/10.1177/1098214009348327 DOI: https://doi.org/10.1177/1098214009348327
Vreeman, R. C., Nyandiko, W. M., Ayaya, S. O., Walumbe, E. G., & Inui, T. S. (2014). Cognitive interviewing for cross-cultural adaptation of pediatric antiretroviral therapy adherence measurement items. International Journal of Behavioral Medicine, 21(1), 186-196. https://doi.org/10.1007/s12529-012-9283-9 DOI: https://doi.org/10.1007/s12529-012-9283-9
Willis G. B. (1999). Cognitive interviewing: A "How To" guide. Proceedings from the 1999 Meeting of the American Statistical Association. https://www.hkr.se/contentassets/9ed7b1b3997e4bf4baa8d4eceed5cd87/gordonwillis.pdf
Willis, G.B. (2005). Cognitive interviewing: A tool for improving survey design. Sage Publications, Inc. https://doi.org/10.1037/e538062007-001 DOI: https://doi.org/10.1037/e538062007-001
Willis, G. B., & Artino, A. R. Jr. (2013). What do our respondents think we're asking? Using cognitive interviewing to improve medical education surveys. Journal of Graduate Medical Education, 5(3), 353-356. https://doi.org/10.4300/JGME-D-13-00154.1 DOI: https://doi.org/10.4300/JGME-D-13-00154.1
Willis, G. B., & Miller, K. (2011). Cross-cultural cognitive interviewing: Seeking comparability and enhancing understanding. Field Methods, 23(4), 331-341. https://doi.org/10.1177/1525822X11416092 DOI: https://doi.org/10.1177/1525822X11416092
Woolley, M. E., Bowen, G. L., & Bowen, N. K. (2004). Cognitive pretesting and the developmental validity of child self-report instruments: Theory and applications. Research on Social Work Practice, 14(3), 191-200. https://doi.org/10.1177/1049731503257882 DOI: https://doi.org/10.1177/1049731503257882
Zeldin, S., Bestul, L., & Powers, J. (2012). Youth-adult partnerships in evaluation (Y-AP/E): A resource guide for translating research into practice. ACT for Youth Center of Excellence, Cornell University. http://actforyouth.net/resources/n/n_yap-eval-guide.pdf