Children’s Opinion of Retrospective Pre-Post ‘Then-Test’ Survey Validity
Main Article Content
Abstract
Background: Over the past forty years there have been a number of studies conducted to compare traditional pre-post surveys (pretest-posttests; administered in two stages before and after an intervention) with retrospective pre-posts (thentests or pre-then-post-tests; administered after intervention only, with participants asked to reflect back to complete the ‘pre’ retrospectively). These previous studies have been with adult respondents and overwhelmingly quantitative.
Purpose: This paper examines children’s perspectives regarding traditional and retrospective pre-post self-report subjective surveys.
Setting: A school-based program run by a community services organisation in southeastern Melbourne.
Intervention: Both pre-post survey types were administered to sixty children attending a pro-social skills group run by a community services organisation in southeast suburban Melbourne.
Research Design: Twenty children participated in eight small focus groups after completing the surveys. Each focus group was guided by three semi-structured questions, ran for 10-15 minutes, and had 2-3 participants. This research included an observation component as the researcher was present at the final session when the post surveys were completed. The research also utilised the quantitative findings from the surveys to check alignment with findings from the extant literature.
Data Collection and Analysis: Focus groups & qualitative analysis
Findings: The traditional and retrospective surveys confirm that the commonly recorded phenomenon of response shift in adults also occurs with children. Children comment that they prefer the retrospective test, identifying concerns that support theories discussed in the extant literature such as experience limitation, impression management, implicit theory of change, and memory recall.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Allen, J. and Nimon, K. (2007). Retrospective pretest: A practical technique for professional development evaluation, Journal of Industrial Teacher Education, 44(3): 27-42.
Bhanji, F., Gottesman, R., de Grave, W., Steinert, Y. and Winer, L. (2012). The retrospective pre-post: A practical method to evaluation learning from an educational program, Academic Emergency Medicine, 19(12), 189-194. https://doi.org/10.1111/j.1553-2712.2011.01270.x DOI: https://doi.org/10.1111/j.1553-2712.2011.01270.x
Blome, C. and Augustin, M. (2015). Measuring change in quality of life: Bias in prospective and retrospective evaluation, Value in Health, 18(1), 110-115. https://doi.org/10.1016/j.jval.2014.10.007 DOI: https://doi.org/10.1016/j.jval.2014.10.007
Campbell, D. and Stanley, J. (1966). Experimental and quasi-experimental designs for research, Boston: Houghton Mifflin Company.
Drennan, J. and Hyde, A. (2008). Controlling response shift bias: The use of the retrospective pre-test design in the evaluation of a master's programme, Assessment & Evaluation in Higher Education, 33(6), 699-709. https://doi.org/10.1080/02602930701773026 DOI: https://doi.org/10.1080/02602930701773026
Eton, D. (2010). Why we need response shift: An appeal to functionalism, Quality of Life Research, 19, 929-930. https://doi.org/10.1007/s11136-010-9684-7 DOI: https://doi.org/10.1007/s11136-010-9684-7
Greig, A., Taylor, J. and MacKay, T. (2013). Doing research with children: A practical guide (3rd ed.), Thousand Oaks (CA): Sage Publications. https://doi.org/10.4135/9781526402219 DOI: https://doi.org/10.4135/9781526402219
Harris, V., Visconti, B., Sengupta, P. and Hinton, G. (2018). Justification for use of the pre-test then retrospective pre-then-post-test evaluation in couple and relationship education, Southeastern Council on Family Relations Conference: Families of Tomorrow: The Intersection of Technology, Theory & Practice, Baton Rouge (LA), March 8-10.
Henry, D., Tolan, P., Gorman-Smith, D. and Schoeny, M. (2017). Alternatives to randomized control trial designs for community-based prevention evaluation, Prevention Science, 18(6), 671-680. https://doi.org/10.1007/s11121-016-0706-8 DOI: https://doi.org/10.1007/s11121-016-0706-8
Hill, L. and Betz, D. (2005). Revisiting the retrospective pretest, American Journal of Evaluation, 26(4), 501-507. https://doi.org/10.1177/1098214005281356 DOI: https://doi.org/10.1177/1098214005281356
Hoogstraten, J. (1982). The retrospective pretest in an educational training context. Journal of Experimental Education, 50(4), 200-204. https://doi.org/10.1080/00220973.1982.11011824 DOI: https://doi.org/10.1080/00220973.1982.11011824
Howard, G. (1980). Response shift bias: A problem in evaluating interventions with pre/post self-reports, Evaluation Review, 4(1), 93-106. https://doi.org/10.1177/0193841X8000400105 DOI: https://doi.org/10.1177/0193841X8000400105
Howard, G., Ralph, K., Gulanick, N., Maxwell, S. and Gerber, S. (1979). Internal invalidity in pretest-posttest self-report evaluations and a re-evaluation of retrospective pretests, Applied Psychological Measurement, 3(1), 1-23. https://doi.org/10.1177/014662167900300101 DOI: https://doi.org/10.1177/014662167900300101
Kelly, L. (2017). Ethics and evaluative consultations with children in small to mid-sized Australian non-government organisations, Evaluation Journal of Australasia, 17(1), 4-11. https://doi.org/10.1177/1035719X1701700102 DOI: https://doi.org/10.1177/1035719X1701700102
Kelly, L. and Smith, K. (2017). Children as capable evaluators: Evolving conceptualizations of childhood in NGO practice settings, Child & Family Social Work, 22(2), 853-861. https://doi.org/10.1111/cfs.12304 DOI: https://doi.org/10.1111/cfs.12304
Kelly, L. (2013). Conducting focus groups with child participants, Developing Practice, 36, 78-82.
Lam, T. and Bengo, P. (2003). A comparison of three retrospective self-reporting methods of measuring change in instructional practice, American Journal of Evaluation, 24(1), 65-80. https://doi.org/10.1177/109821400302400106 DOI: https://doi.org/10.1177/109821400302400106
Lamb, T. (2005). The retrospective pretest: An imperfect but useful tool, The Evaluation Exchange, 11(2), 18.
Lindberg, P., Netter, P., Koller, M., Steinger, B. and Klinkhammer-Schalke, M. (2017). Breast cancer survivors' recollection of their quality of life: Identifying determinants of recall bias in a longitudinal population-based trial, PLoS ONE 12(2), 1-14. https://doi.org/10.1371/journal.pone.0171519 DOI: https://doi.org/10.1371/journal.pone.0171519
Marshall, J., Higginbotham, B., Harris, V. & Lee, T. (2007). Assessing program outcomes: Rationale and benefits of posttest-then-retrospective-pretest designs, Journal of Youth Development, 2(1), Article 0701RS001. https://doi.org/10.5195/jyd.2007.366 DOI: https://doi.org/10.5195/jyd.2007.366
McKenna, M., Kear, D. and Ellsworth, R. (1995). Children's attitudes toward reading: A national survey, Reading Research Quarterly, 30(4), 934-956. https://doi.org/10.2307/748205 DOI: https://doi.org/10.2307/748205
Mueller, C. (2015). Evaluating the effectiveness of website content features using retrospective pretest methodology: An experimental test. Evaluation Review, 39(3), 283-307. https://doi.org/10.1177/0193841X15582142 DOI: https://doi.org/10.1177/0193841X15582142
Nieuwkerk, P., Tollenaar, M., Oort, F. and Sprangers, M. (2007). Are retrospective measures of change in quality of life more valid than prospective measures? Medical Care, 45(3), 199-205. https://doi.org/10.1097/01.mlr.0000246613.49214.46 DOI: https://doi.org/10.1097/01.mlr.0000246613.49214.46
Nimon, K., Zigarmi, D. and Allen, J. (2011). Measures of program effectiveness based on retrospective pretest data: Are all created equal? American Journal of Evaluation 32(1), 8-28. https://doi.org/10.1177/1098214010378354 DOI: https://doi.org/10.1177/1098214010378354
Nimon, K. (2014). Explaining differences between retrospective and traditional pretest self-assessments: Competing theories and empirical evidence, International Journal of Research and Method in Education, 37(3), 256-269. https://doi.org/10.1080/1743727X.2013.820644 DOI: https://doi.org/10.1080/1743727X.2013.820644
Nolte, S., Elsworth, G., Sinclair, A. and Osborne, R. (2009). Tests of measurement invariance failed to support the application of the 'then-test'. Journal of Clinical Epidemiology, 62(11), 1173-1180. https://doi.org/10.1016/j.jclinepi.2009.01.021 DOI: https://doi.org/10.1016/j.jclinepi.2009.01.021
Nolte, S., Elsworth, G., Sinclair, A. and Osborne, R. (2012). The inclusion of 'then-test' questions in post-test questionnaires alters post-test responses: A randomized study of bias in health program evaluation, Quality of Life Research, 21(3), 487-494. https://doi.org/10.1007/s11136-011-9952-1 DOI: https://doi.org/10.1007/s11136-011-9952-1
Norman, G. (2003). Hi! How are you? Response shift, implicit theories and differing epistemologies, Quality of Life Research, 12(3), 239-249. https://doi.org/10.1023/A:1023211129926 DOI: https://doi.org/10.1023/A:1023211129926
Pelfrey, W. Snr. and Pelfrey W. Jr. (2009). Curriculum evaluation and revision in a nascent field: The utility of the retrospective pretest-posttest model in a homeland security program of study, Evaluation Review, 33(1), 54-82. https://doi.org/10.1177/0193841X08327578 DOI: https://doi.org/10.1177/0193841X08327578
Piwowar, V. and Thiel, F. (2014). Evaluating response shift in training evaluation: Comparing the retrospective pretest with an adapted measurement invariance approach in a classroom management program. Evaluation Review, 38(5), 420-448. https://doi.org/10.1177/0193841X14546932 DOI: https://doi.org/10.1177/0193841X14546932
Pratt, C., McGuigan, W. & Katzev, A., 2000. Measuring program outcomes: Using retrospective pretest methodology, American Journal of Evaluation, 21, 341-349. https://doi.org/10.1177/109821400002100305 DOI: https://doi.org/10.1177/109821400002100305
Rees, G., Goswami, H. and Bradshaw, J. (2010). Developing an index of children's subjective wellbeing, The Children's Society, London.
Rohs, F. (2002). Improving the evaluation of leadership programs: Control response shift, Journal of Leadership Education, 1(2), 1-12. https://doi.org/10.12806/V1/I2/RF3 DOI: https://doi.org/10.12806/V1/I2/RF3
Ross, M. (1989). Relation of implicit theories to the construction of personal histories. Psychological Review, 96(2): 341-357. https://doi.org/10.1037//0033-295X.96.2.341 DOI: https://doi.org/10.1037//0033-295X.96.2.341
Ross, M. and Conway, M. (1986). Remembering one's own past: The construction of personal histories. In R. Sorrentino and E. Higgins (Eds.), Handbook of motivation and cognition, (pp. 122-144). New York: Guilford Press.
Schwarz, N. (2007). Retrospective and concurrent self-reports: The rationale for real-time data capture. In A. Stone, S. Shiffman, A. Atienza and L. Nebeling (Eds.), The science of real-time data capture, (pp. 11-26). New York: Oxford University Press. https://doi.org/10.1093/oso/9780195178715.003.0002 DOI: https://doi.org/10.1093/oso/9780195178715.003.0002
Schwartz, C. and Sprangers, M. (2010). Guidelines for improving the stringency of response shift research using the thentest. Quality of Life Research, 19(4), 455-464. https://doi.org/10.1007/s11136-010-9585-9 DOI: https://doi.org/10.1007/s11136-010-9585-9
Sprangers, M. and Hoogstraten, J. (1988). On delay and reassessment of retrospective preratings. Journal of Experimental Education, 56(3), 148-153. https://doi.org/10.1080/00220973.1988.10806480 DOI: https://doi.org/10.1080/00220973.1988.10806480
Sprangers, M. (1989). Response shift bias in program evaluation, Impact Assessment, 7(2-3), 153-166. https://doi.org/10.1080/07349165.1989.9726018 DOI: https://doi.org/10.1080/07349165.1989.9726018
Sprangers, M. and Hoogstraten, J. (1991). Subject bias in three self-report measures of change, Methodika, 5, 1-13.
Sullivan, L. and Haley, K. (2009). Using a retrospective pretest to measure learning in professional development programs, Community College Journal of Research and Practice, 33(3-4), 346-362. https://doi.org/10.1080/10668920802565052 DOI: https://doi.org/10.1080/10668920802565052
Taminiau-Bloem, E., Schwartz, C., van Zuuren, F., Koeneman, M., Visser, M., Tishelman, C., Koning, C. and Sprangers, M. (2016). Using a retrospective pretest instead of a conventional pretest is replacing biases: A qualitative study of cognitive processes underlying responses to thentest items, Quality of Life Research, 25, 1327-1337. https://doi.org/10.1007/s11136-015-1175-4 DOI: https://doi.org/10.1007/s11136-015-1175-4
Taylor, P., Russ-Eft, D. and Chan, D. (2003). The impact of alternative rating sources and retrospective pretests on training effect sizes. 5th Australian Industrial and Organisational Psychology Conference Proceedings, Australian Psychological Society, Melbourne, June 26-29.
Taylor, P., Russ-Eft, D. and Taylor, H. (2009). Gilding the outcome by tarnishing the past: Inflationary biases in retrospective pretest. American Journal of Evaluation 30(1): 31-34. https://doi.org/10.1177/1098214008328517 DOI: https://doi.org/10.1177/1098214008328517
Terborg, J. and Davis, G. (1980). Evaluation of a new method for assessing change to planned job redesign as applied to Hackman and Oldham's job characteristic model, Technical Report 80-6, Graduate School of Management, University of Oregon, Eugene (OR). https://doi.org/10.1037/e594792009-001 DOI: https://doi.org/10.1037/e594792009-001
Wolfson, A. and Carskadon, M. (2003). Understanding adolescent's sleep patterns and school performance: A critical appraisal, Sleep Medicine Reviews, 7(6), 491-506. https://doi.org/10.1016/S1087-0792(03)90003-7 DOI: https://doi.org/10.1016/S1087-0792(03)90003-7