The Role of Random Allocation in Randomized Controlled Trials: Distinguishing Selection Bias from Baseline Imbalance
Main Article Content
Abstract
Background: This paper addresses one threat to the internal validity of a randomized controlled trial (RCT), selection bias. Many authors argue that random allocation is used to ensure baseline equality between study conditions in a given study and that statistically significant differences at pretest mean that randomisation has failed.
Purpose: The purpose of this study was to clarify the role of random allocation in an RCT study. Is the role of random allocation to protect against selection bias? And does it have a further role, namely to ensure baseline equality and the absence of statistically significant differences between study conditions at pretest?
Setting: The participants for this study were 229 children in 1st and 2nd grade and data were collected as part of an RCT evaluation of a volunteer reading programme piloted in Ireland, Wizards of Words (WoW).
Intervention: Not applicable.
Research design: The allocation procedure adopted in this study was stratified and blocked random allocation.
Data collection and analysis: Data were collected using standardised and criterion-referenced tests of reading ability. Data were collected by qualified Speech and Language Therapists. Independent-samples t-tests were used to analyse pretest data.
Findings: The role of random allocation is to protect against selection bias, and statistically significant baseline differences can result even when random allocation has been successful. Whether or not random allocation has been successful is determined by the generation of the random allocation sequence and the steps taken to ensure its concealment. The size of differences between study conditions at pretest can be important for the analysis of posttest data but does not by itself determine whether random allocation was successful. In addition, there are serious concerns about the appropriateness of tests of significance when comparing two study conditions at pretest.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
Funding data
References
Altman, D. G. (1985). Comparability of randomized groups. The Statistician, 34, 125-136. https://doi.org/10.2307/2987510 DOI: https://doi.org/10.2307/2987510
Altman, D. G. (1991). Randomization: Essential for reducing bias. British Medical Journal, 302, 1481-1482. https://doi.org/10.1136/bmj.302.6791.1481 DOI: https://doi.org/10.1136/bmj.302.6791.1481
Altman, D. G., Moher, D., Schulz, K. F., Egger, M., Davidoff, F., Elbourne, D., Gotsche, P., & Lang, T. (2001). The CONSORT statement: Revised recommendations for improving the quality of reports of parallel group randomized trials. Annals of Internal Medicine, 134(8), 657-662. https://doi.org/10.7326/0003-4819-134-8-200104170-00011 DOI: https://doi.org/10.7326/0003-4819-134-8-200104170-00011
Assmann, S. F., Pocock, S. J., Enos, L. E., & Kasten, L. E. (2000). Subgroup analysis and other (mis)uses of baseline data in clinical trials. Lancet, 335(9235), 1064-69. https://doi.org/10.1016/S0140-6736(00)02039-0 DOI: https://doi.org/10.1016/S0140-6736(00)02039-0
Axford, N., Morpeth, L., Little, M., & Berry, V. (2008). Linking prevention science and community engagement: a case study of the Ireland Disadvantaged Children and Youth Programme. Journal of Children's Services, 3(2), 40-54. https://doi.org/10.1108/17466660200800011 DOI: https://doi.org/10.1108/17466660200800011
Baker, S., Gersten, R., & Keating, T. (2000). When less may be more: A 2-year longitudinal evaluation of a volunteer tutoring program requiring minimal training. Reading Research Quarterly, 35(4), 494-519. https://doi.org/10.1598/RRQ.35.4.3 DOI: https://doi.org/10.1598/RRQ.35.4.3
Beller, E. M., Gebski, V., & Keech, A. C. (2002). Randomization in clinical trials. Medical Journal of Australia, 177(10), 565-567. https://doi.org/10.5694/j.1326-5377.2002.tb04955.x DOI: https://doi.org/10.5694/j.1326-5377.2002.tb04955.x
Berger, V. W., & Weinstein, S. (2004). Ensuring the comparability of comparison groups: is randomization enough? Controlled Clinical Trials, 25(5), 515-524. https://doi.org/10.1016/j.cct.2004.04.001 DOI: https://doi.org/10.1016/j.cct.2004.04.001
Boruch, R., Weisburd, D., Turner III, H. M., Karpyn, A., & Littell, J. (2009). Randomized Controlled Trials for Evaluation and Planning. In L. Bickman & D. J. Rog (Eds.), The Sage Handbook of Applied Social Research. Second edition (pp. 147-181). London: Sage. https://doi.org/10.4135/9781483348858.n5 DOI: https://doi.org/10.4135/9781483348858.n5
Cohen, J. (1988). Statistical Power Analysis for the Behavioural Sciences. Second edition. New Jersey: Lawrence Erlbaum Associates.
Dunn, L., Dunn, L., Whetton, C., & Burley, J. (1997) The British Picture Vocabulary Scale. Second edition. London: GL Assessment.
Field, A. (2009). Discovering Statistics Using SPSS. Third edition (London: Sage).
Ghate, D. (2001). Community-based evaluations in the UK: Scientific concerns and practical constraints. Children and Society, 15(1), 23-32. https://doi.org/10.1111/j.1099-0860.2001.tb00200.x DOI: https://doi.org/10.1111/j.1099-0860.2001.tb00200.x
Government Social Research Unit (GSR) (2007). Magenta Book Background Papers. Paper 7: why do social experiments? London: HM Treasury. Available at http://www.civilservice.gov.uk/Assets/chap_6_magenta_tcm6-8609.pdf (accessed January 2011)
Hatcher, P. J., Hulme, C., Miles, J., Carroll, J., Hatcher, J., Gibbs, S., Smith G., Bowyer-Crane, C., & Snowling, M. (2006). Efficacy of small group reading intervention for beginning readers with reading-delay: A randomized controlled trial. Journal of Child Psychology and Psychiatry, 47(8), 820-827. https://doi.org/10.1111/j.1469-7610.2005.01559.x DOI: https://doi.org/10.1111/j.1469-7610.2005.01559.x
Hutchings, J., Bywater, T., Eames, C., & Martin, P. (2008). Implementing child mental health interventions in service settings: Lessons from three pragmatic randomised controlled trials in Wales. Journal of Children's Services, 3(2), 17-27. https://doi.org/10.1108/17466660200800009 DOI: https://doi.org/10.1108/17466660200800009
Institute of Education Sciences (2011). What Works Clearinghouse. Procedures and Standards Handbook. Version 2.1. Available at: http://ies.ed.gov/ncee/wwc/documentsum.aspx?sid=19
Jaded, A. (1998). Randomised Controlled Trials: A User's Guide. London: BMJ Books.
Jones, M., Gebski, V., Onslow, M., & Packman, A. (2001). Design of randomized controlled trials: Principles and methods applied to a treatment for early stuttering. Journal of Fluency Disorders, 26(4), 247-267. https://doi.org/10.1016/S0094-730X(01)00108-5 DOI: https://doi.org/10.1016/S0094-730X(01)00108-5
Juni, P., Altman, D. G., & Egger, M. (2001). Assessing the quality of controlled clinical trials. British Medical Journal, 323(7303), 42-46. https://doi.org/10.1136/bmj.323.7303.42 DOI: https://doi.org/10.1136/bmj.323.7303.42
Kernan, W. N., Viscoli, C. M., Makuch, R. W., Brass, L. M., & Horwitz, R. I. (1999). Stratified randomization for clinical trials. Journal of Clinical Epidemiology, 52(1), 19-26. https://doi.org/10.1016/S0895-4356(98)00138-3 DOI: https://doi.org/10.1016/S0895-4356(98)00138-3
Lachin, J. M., Matts, J. P., & Wei, L. J. (1988). Randomization in clinical trials: Conclusions and recommendations. Controlled Clinical Trials, 9(4), 365-374. https://doi.org/10.1016/0197-2456(88)90049-9 DOI: https://doi.org/10.1016/0197-2456(88)90049-9
McIlroy, D. (2010). The fundamental importance of baseline comparisons in a clinical trial. The Journal of Thoracic and Cardiovascular Surgery, 139(3), 801-801. https://doi.org/10.1016/j.jtcvs.2009.10.058 DOI: https://doi.org/10.1016/j.jtcvs.2009.10.058
Morrison, K. (2001). Randomised controlled trials for evidence-based education: Some problems in judging "What Works." Evaluation and Research in Education, 15(2), 69-83. https://doi.org/10.1080/09500790108666984 DOI: https://doi.org/10.1080/09500790108666984
Morrow-Howell, N., Jonson-Reid, M., McCrary, S., Lee, Y. S., & Spitznagel, E. (2009). Evaluation of Experience Corps Student Reading Outcomes. Available at: http://www.mathematica.mpr.com/publications/SearchList2.aspx?jumpsrch=yes&txtSearch=%22Susan%20Sprachman%22%20or%20%22S.%20Sprachman%22 (accessed March 2010).
Oakley, A., Strange, V., Toroyan, T., Wiggins, M., Roberts, I., & Stephenson, J. (2003). Using random allocation to evaluate social interventions: Three recent U.K. examples. The ANNALS of the American Academy of Political and Social Science, 589(1), 170-189. https://doi.org/10.1177/0002716203254765 DOI: https://doi.org/10.1177/0002716203254765
Pawson, R., & Tilley, N. (1997). Realistic Evaluation. London: Sage.
Pocock, S. J., Assmann, S. E., Enos, L. E., & Kasten, L. E. (2002). Subgroup analysis, covariate adjustment and baseline comparisons in clinical trial reporting: current practice and problems. Statistics in Medicine, 21(19), 2917-2930. https://doi.org/10.1002/sim.1296 DOI: https://doi.org/10.1002/sim.1296
Pullen, P. C., Lane, H. B., & Monaghan, M. C. (2004) Effects of a volunteer tutoring model on the early literacy development of struggling first grade students. Reading Research and Instruction, 43(4), 21-40. https://doi.org/10.1080/19388070409558415 DOI: https://doi.org/10.1080/19388070409558415
Ransohoff, D. F. (2005). Bias as a threat to the validity of cancer molecular-marker research. Nature Reviews, 5(2), 142-149. https://doi.org/10.1038/nrc1550 DOI: https://doi.org/10.1038/nrc1550
Ritter, G., & Maynard, R. A. (2008). Using the right design to get the 'wrong' answer? Results of a random assignment evaluation of a volunteer tutoring programme. Journal of Children's Services, 3(2), 4-16. https://doi.org/10.1108/17466660200800008 DOI: https://doi.org/10.1108/17466660200800008
Ritter, G., & Holley, M. (2008). Lessons for conducting random assignment in schools. Journal of Children's Services, 3(2), 28-39. https://doi.org/10.1108/17466660200800010 DOI: https://doi.org/10.1108/17466660200800010
Ritter, G., Barnett, J. H., Denny, G. S., & Albin, G. R. (2009). The effectiveness of volunteer tutoring programs for elementary and middle school students: A meta-analysis. Review of Educational Research, 79(1), 3-38. https://doi.org/10.3102/0034654308325690 DOI: https://doi.org/10.3102/0034654308325690
Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A Systematic Approach. Seventh edition. London: Sage.
Russell, D. W., Kahn, J. H., Spoth, R., & Altmaier, E. M. (1998). Analyzing data from experimental studies: A latent variable structural equation modeling approach. Journal of Counseling Psychology, 45(1), 18-29. https://doi.org/10.1037//0022-0167.45.1.18 DOI: https://doi.org/10.1037//0022-0167.45.1.18
Savage, R., Carless, S., Stuart, M. (2003). The effects of rime- and phoneme-based teaching delivered by Learning Support assistants. Journal of Research in Reading, 26(3), 211-233. https://doi.org/10.1111/1467-9817.00199 DOI: https://doi.org/10.1111/1467-9817.00199
Schulz, K. F., & Grimes, D. A. (2002). Generation of allocation sequences in randomized trials: chance, not choice. The Lancet, 359(9305), 515-519. https://doi.org/10.1016/S0140-6736(02)07683-3 DOI: https://doi.org/10.1016/S0140-6736(02)07683-3
Senn, S. J., (1994) Testing for baseline balance in clinical trials. Statistics in Medicine, 13(17), 1715-1726. https://doi.org/10.1002/sim.4780131703 DOI: https://doi.org/10.1002/sim.4780131703
Senn, S. J. (1995). Base logic: Tests of imbalance in randomized clinical trials. Clinical Research and Regulatory Affairs, 12(3), 171-182. https://doi.org/10.3109/10601339509019426 DOI: https://doi.org/10.3109/10601339509019426
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. New York: Houghton Mifflin Co.
Simon, R. (1979). Restricted randomization designs in clinical trials. Biometrics, 35, 503-512. https://doi.org/10.2307/2530354 DOI: https://doi.org/10.2307/2530354
Slavin, R. (2008). What Works? issues in synthesizing educational program evaluations. Educational Researcher, 37(1), 5-14. https://doi.org/10.3102/0013189X08314117 DOI: https://doi.org/10.3102/0013189X08314117
Snowling, M., Stothard, S., Clarke, P., Bowyer-Crane, C., Harrington, A., Truelove, E., & Hulme, C. (2009). York Assessment of Reading for Comprehension. University of York: Cantre for Reading and Language.
Society for Prevention Research (2004). Standards of Evidence: Criteria for efficacy, effectiveness and dissemination. Available at: http://www.preventionresearch.org/StandardsofEvidencebook.pdf
Tierney, J. P., Grossman, J. B., & Resch, N. L. (1995). Making a Difference: An Impact Study of Big Brothers Big Sisters. Public/Private Ventures. Available at: http://www.ppv.org/ppv/publication.asp?search_id=7&publication_id=111§ion_id=0
Torgerson, C. J. (2001). The need for randomized controlled trials in educational research. British Journal of Educational Studies, 49(3), 316-328. https://doi.org/10.1111/1467-8527.t01-1-00178 DOI: https://doi.org/10.1111/1467-8527.t01-1-00178
Torgerson, D., & Torgerson, C. (2003). Avoiding bias in randomized controlled trials in education research. British Journal of Educational Studies, 51(1), 36-45. https://doi.org/10.1111/1467-8527.t01-2-00223 DOI: https://doi.org/10.1111/1467-8527.t01-2-00223
Wasik, B. A. (1998). Volunteer tutoring programs in reading: A review. Reading Research Quarterly, 33(3), 266-292. https://doi.org/10.1598/RRQ.33.3.2 DOI: https://doi.org/10.1598/RRQ.33.3.2
Wasik, B. A. & Slavin, R. (1993). Preventing early reading failure with one-to-one tutoring: A review of five programs. Reading Research Quarterly, 28(2), 179-200. https://doi.org/10.2307/747888 DOI: https://doi.org/10.2307/747888
Yu, L-M., Chan, A-W., Hopewell, S., Deeks, J. J., & Altman, D. G. (2010). Reporting on covariate adjustment in randomized controlled trials before and after revision of the 2010 CONSORT statement: a literature review. Trials, 11, 59 (Open Access). https://doi.org/10.1186/1745-6215-11-59 DOI: https://doi.org/10.1186/1745-6215-11-59