Comparison Group Identification for Difficult-to-Evaluate Populations: Lessons from Evaluating 1n10 LGBTQ Youth Support Services
Main Article Content
Abstract
The evaluation literature on Lesbian, Gay, Bisexual, Transgender, and Queer (LGBTQ) youth programs is largely absent of quantitative studies, likely in part due to the challenge of using experimental or quasi-experimental evaluation designs. This paper proposes the creative use of a national data set to overcome the problem of estimating a counterfactual for this population. In addition to discussing lessons from this approach, we describe the program and its impacts. Evidence suggests that the program under study—1n10, a local support group for LGBTQ youth in the Phoenix, AZ, metropolitan area—draws a relatively more disadvantaged group than the national average but that their levels of suicide ideation are lower. By addressing key threats to the internal validity of evaluation designs, we rule out plausible rival explanations for program impacts.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Aiken, L. S., West, S. G., Schwalm, D. E., Carroll, J., & Hsuing, S. (1998). Comparison of a randomized and two quasi- experimental designs in a single outcome evaluation: Efficacy of a university-level remedial writing program. Evaluation Review, 22(4), 207-244. DOI: https://doi.org/10.1177/0193841X9802200203
https://doi.org/10.1177/0193841X9802200203 DOI: https://doi.org/10.1177/0193841X9802200203
Bamberger, M., Rugh, J., Church, M., & Fort, L. (2003). Shoestring evaluation: Designing impact evaluations under budget, time and data constraints. American Journal of Evaluation, 25(1), 5-37. DOI: https://doi.org/10.1177/109821400402500102
https://doi.org/10.1177/109821400402500102 DOI: https://doi.org/10.1177/109821400402500102
Beals, J., Spicer, P., Mitchell, C., Novins, D., Manson, S., & the Al-SUPERPFP Team. (2003). Racial disparities in alcohol use: Comparison of two american indian reservation populations with national Data. American Journal of Public Health, 93 (10), 1683-1685. DOI: https://doi.org/10.2105/AJPH.93.10.1683
https://doi.org/10.2105/AJPH.93.10.1683 DOI: https://doi.org/10.2105/AJPH.93.10.1683
Bell, A., & Weinberg, M. (1978). Homosexualities: A study of diversity among men and women. New York: Simon and Shuster.
Bloom, H., Michalopoulos, C., Hill, C.,& Ying, L. (2002). Can nonexperimental comparison group methods match the findings from a random assignment evaluation of mandatory welfare-to-work programs?, Working Paper, Manpower Demonstration Research Corporation.
Cochran, W. G., & Chambers, S. P. (1965). The planning of observational studies of human populations. Journal of the Royal Statistical Society. Series A, 128 (2), 234-266. DOI: https://doi.org/10.2307/2344179
https://doi.org/10.2307/2344179 DOI: https://doi.org/10.2307/2344179
Cook, T. D., Shadish, W. R., & Wong, V. C. (2005). Within study comparisons of experiments and non-experiments: Can they help decide on evaluation policy? (pp. 1-46): French Economic Society Meeting on Program Evaluation.
D'Agostino, R. B., & Kwan, H. (1995). Measuring effectiveness: What to expect without a randomized control group. Medical Care, 33(Supplement), AS95-AS105.
D'Agostino, R. B., & Rubin, D. B. (2000). Estimating and using propensity scores with partially missing data. Journal of the American Statistical Association, 95(451), 749-759. DOI: https://doi.org/10.1080/01621459.2000.10474263
https://doi.org/10.1080/01621459.2000.10474263 DOI: https://doi.org/10.1080/01621459.2000.10474263
D'Augelli, A., & Hershberger, S. (1993). Lesbian, gay, and bisexual youth in community settings: Personal challenges and mental health problems. American Journal of Community Psychology, 21, 421-448. DOI: https://doi.org/10.1007/BF00942151
https://doi.org/10.1007/BF00942151 DOI: https://doi.org/10.1007/BF00942151
DeCrescenzo, T. (1992). The brave new world of gay and lesbian youth. In B. Berzon (Ed.), Positively Gay. Berkeley, CA: Celestial Arts.
DuRant, R. H., Krowchuk, D., & Sinal, S. (1998). Victimization, use of violence, and drug use at school among male adolescents who engage in same-sex sexual behavior Journal of Pediatrics, 133(1), 113-118. DOI: https://doi.org/10.1016/S0022-3476(98)70189-1
https://doi.org/10.1016/S0022-3476(98)70189-1 DOI: https://doi.org/10.1016/S0022-3476(98)70189-1
Faulkner, A. H., & Cranston, K. (1998). Correlates of same-sex sexual behavior in a random sample of Massachusetts high school students. American Journal of Public Health, 88(2), 262-266. DOI: https://doi.org/10.2105/AJPH.88.2.262
https://doi.org/10.2105/AJPH.88.2.262 DOI: https://doi.org/10.2105/AJPH.88.2.262
Garofalo, R., Wolf, R. C., Kessel, S., Palfrey, S. J., & DuRant, R. H. (1998). The association between health-risk behaviors and sexual orientation among a school-based sample of adolescents. Pediatrics, 101, 895-902. DOI: https://doi.org/10.1542/peds.101.5.895
https://doi.org/10.1542/peds.101.5.895 DOI: https://doi.org/10.1542/peds.101.5.895
Grossman, A. H., & Kerner, M. S. (1998). Self Esteem and supportiveness as predictors of emotional distress in gay male and lesbian youth. Journal of Homosexuality, 35(2), 25-39 DOI: https://doi.org/10.1300/J082v35n02_02
https://doi.org/10.1300/J082v35n02_02 DOI: https://doi.org/10.1300/J082v35n02_02
Hammelman, T. (1993). Gay and lesbian youth: Contributing factors to serious attempts or considerations of suicide. Journal of Gay and Lesbian Psychotherapy, 2(1), 77-89. DOI: https://doi.org/10.1300/J236v02n01_06
https://doi.org/10.1300/J236v02n01_06 DOI: https://doi.org/10.1300/J236v02n01_06
Herdt, G., & Boxer, A. (1993). Children of Horizons. Boston: Beacon Press.
Joyner, K., & Udry, J. R. (2000). You don't bring me anything but down: Adolescent romantic relationships and depression. Journal of Health and Social Behavior, 41(December), 369-391. DOI: https://doi.org/10.2307/2676292
https://doi.org/10.2307/2676292 DOI: https://doi.org/10.2307/2676292
Kourany, R. (1987). Suicide among homosexual adolescents. Journal of Homosexuality, 13, 111- 117. DOI: https://doi.org/10.1300/J082v13n04_07
https://doi.org/10.1300/J082v13n04_07 DOI: https://doi.org/10.1300/J082v13n04_07
Kruks, G. (1991). Gay and lesbian homeless street youth: Special issues and concerns. Journal of Adolescent Health, 12, 515-518. DOI: https://doi.org/10.1016/0197-0070(91)90080-6
https://doi.org/10.1016/0197-0070(91)90080-6 DOI: https://doi.org/10.1016/0197-0070(91)90080-6
Lee, C. (2002). The impact of belonging to a high school gay/straight alliance. The High School Journal, Feb/Mar 2002. DOI: https://doi.org/10.1353/hsj.2002.0005
https://doi.org/10.1353/hsj.2002.0005 DOI: https://doi.org/10.1353/hsj.2002.0005
LeVay, S., & Nonas, E. (1995). City of FriendsA A portrait of the gay and lesbian community in America. Cambridge, MA: The MIT Press.
Lock, J., & Steiner, H. (1998). Gay, Lesbian, and Bisexual Youth Risks for Emotional, Physical, and Social Problems: Results from a community-based survey. Journal of the American Academy of Child and Adolescent Psychiatry, 38(3), 297-304. DOI: https://doi.org/10.1097/00004583-199903000-00017
https://doi.org/10.1097/00004583-199903000-00017 DOI: https://doi.org/10.1097/00004583-199903000-00017
Marsh, H. W. (1998). Simulation study of non- equivalent group-matching and regression- discontinuity designs: Evaluation of gifted and talented programs. Journal of Experimental Education, 66, 163-192. DOI: https://doi.org/10.1080/00220979809601402
https://doi.org/10.1080/00220979809601402 DOI: https://doi.org/10.1080/00220979809601402
Nesmith, A., Burton, D., & Cosgrove, T. (1999). Gay, lesbian, and bisexual youth and young adults: Social support in their own words. Journal of Homosexuality, 37(1), 95-108. DOI: https://doi.org/10.1300/J082v37n01_07
https://doi.org/10.1300/J082v37n01_07 DOI: https://doi.org/10.1300/J082v37n01_07
Noell, J., & Ochs, L. (2001). Relationship of Sexual Orientation to Substance Abuse, Suicidal Ideation, Suicide Attempts, and Other Factors in a Population of Homeless Adolescents. Journal of Adolescent Health, 29(1), 31-36. DOI: https://doi.org/10.1016/S1054-139X(01)00205-1
https://doi.org/10.1016/S1054-139X(01)00205-1 DOI: https://doi.org/10.1016/S1054-139X(01)00205-1
Patton, M. C. (1982). Practical Evaluation. Newbury Park, CA: Sage.
Peck, L. R. (2003). Subgroup Analysis in Social Experiments: Measuring Program Impacts Based on Post-Treatment Choice. American Journal of Evaluation, 24(2), 157-187. DOI: https://doi.org/10.1016/S1098-2140(03)00031-6
https://doi.org/10.1177/109821400302400203 DOI: https://doi.org/10.1177/109821400302400203
Pfeiffer, S., & Strzelecki, B. (1990). Inpatient Psychiatric Treatment of Children and Adolescents: A Review of Outcome Studies. Journal of the American Academy of Child and Adolescent Psychiatry, 29, 847-853. DOI: https://doi.org/10.1097/00004583-199011000-00001
https://doi.org/10.1097/00004583-199011000-00001 DOI: https://doi.org/10.1097/00004583-199011000-00001
Ramfedi, G., Farrow, J., & Deisher, R. (1991). Risk factors for attempted suicide in gay and bisexual yoth. Pediatrics, 87, 869-875. DOI: https://doi.org/10.1542/peds.87.6.869
https://doi.org/10.1542/peds.87.6.869 DOI: https://doi.org/10.1542/peds.87.6.869
Ramfedi, G., French, S., Story, M., Resnick, M., & Blum, R. (1998). The relationship between suicide risk and sexual orientaion: Results of a population-based study. American Journal of Public Health, 88(1), 57-60. DOI: https://doi.org/10.2105/AJPH.88.1.57
https://doi.org/10.2105/AJPH.88.1.57 DOI: https://doi.org/10.2105/AJPH.88.1.57
Rosenbaum, P. R., & Rubin, D. B. (1983). The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika, 70(1), 41-55. DOI: https://doi.org/10.1093/biomet/70.1.41
https://doi.org/10.1093/biomet/70.1.41 DOI: https://doi.org/10.1093/biomet/70.1.41
Rosenbaum, P. R., & Rubin, D. B. (1984). Reducing Bias in Observational Studies Using Subclassification on the Propensity Score. Journal of the American Statistical Association, 79(387), 516-524. DOI: https://doi.org/10.1080/01621459.1984.10478078
https://doi.org/10.1080/01621459.1984.10478078 DOI: https://doi.org/10.1080/01621459.1984.10478078
Rosenberg, M. (1979). Conceiving the self. New York: Basic Books.
Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A Systematic Approach. Thousand Oaks, CA: Sage.
Rubin, D. B., & Thomas, N. (1996). Matching Using Estimated Propensity Scores: Relating Theory to Practice. Biometrics, 52, 249-264. DOI: https://doi.org/10.2307/2533160
https://doi.org/10.2307/2533160 DOI: https://doi.org/10.2307/2533160
Russell, S. T., & Consolacion, T. B. (2003). Adolescent Romance and Emotional Health in the United States: Beyond Binaries. Journal of Clinical and Adolescent Psychology 32(4), 499-508. DOI: https://doi.org/10.1207/S15374424JCCP3204_2
https://doi.org/10.1207/S15374424JCCP3204_2 DOI: https://doi.org/10.1207/S15374424JCCP3204_2
Russell, S. T., Franz, B. T., & Driscoll, A. K. (2001). Same-sex Romantic Attraction and Experiences of Violence in Adolescence. American Journal of Public Health, 91(6), 903- 906. DOI: https://doi.org/10.2105/AJPH.91.6.903
https://doi.org/10.2105/AJPH.91.6.903 DOI: https://doi.org/10.2105/AJPH.91.6.903
Russell, S. T., & Joyner, K. (2001). Adolescent Sexual Orientation and Suicide Risk: DOI: https://doi.org/10.2105/AJPH.91.8.1276
https://doi.org/10.2105/AJPH.91.8.1276 DOI: https://doi.org/10.2105/AJPH.91.8.1276
Evidence from a national study. American Journal of Public Health, 91(8), 1276-1281.
Russell, S. T., & Seif, H. (202). Bisexual female adolescents: A critical review of past research, and results from a national study. Journal of Bisexuality, 2, 73-94. DOI: https://doi.org/10.1300/J159v02n02_05
https://doi.org/10.1300/J159v02n02_05 DOI: https://doi.org/10.1300/J159v02n02_05
Russell, S. T., Seif, H., & Truong, N. L. (2001). School Outcomes of Sexual Minority Youth in the United States: Evidence from a national study. Journal of Adolescence, 24(1), 111-127. DOI: https://doi.org/10.1006/jado.2000.0365
https://doi.org/10.1006/jado.2000.0365 DOI: https://doi.org/10.1006/jado.2000.0365
Saghir, M., & Robins, E. (1973). Male and female homosexuality: A comprehensive investigation. Baltimore: Williams and Wilkins.
Savin-Williams, R. (1990). Gay and lesbian youth: Expressions of identity. New York: Hemisphere Publishing Corp.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.
Shaw, R., Rosati, M., Salzman, P., Coles, C., & McGeary, C. (1997). Effects of adolescent ATOD behaviors and attitudes of a 5-year community partnership. Evaluation and Program Planning, 20 (3), 307-313. DOI: https://doi.org/10.1016/S0149-7189(97)00010-4
https://doi.org/10.1016/S0149-7189(97)00010-4 DOI: https://doi.org/10.1016/S0149-7189(97)00010-4
Shifrin, F., & Solis, M. (1992). Chemical dependency in gay and lesbian youth. Journal of Chemical Dependency Treatment, 5(1), 67-76. DOI: https://doi.org/10.1300/J034v05n01_06
https://doi.org/10.1300/J034v05n01_06 DOI: https://doi.org/10.1300/J034v05n01_06
Smith, H. L. (1997). Matching with Multiple Controls to Estimate Treatment Effects in Observational Studies. Soicological Methodology, 27, 325-353. DOI: https://doi.org/10.1111/1467-9531.271030
https://doi.org/10.1111/1467-9531.271030 DOI: https://doi.org/10.1111/1467-9531.271030
Smith, J. A., & Todd, P. E. (2001). Reconciling Conflicting Evidence on the Performance of Propensity-Score Matching Methods. The American Economic Review, 91(2), 112-118. DOI: https://doi.org/10.1257/aer.91.2.112
https://doi.org/10.1257/aer.91.2.112 DOI: https://doi.org/10.1257/aer.91.2.112
Ury, H. K. (1975). Efficiency of Case-Control Studies with Multiple Controls per Case: Continuos or Dichotomous Data. Biometrics, 31, 643-649. DOI: https://doi.org/10.2307/2529548
https://doi.org/10.2307/2529548 DOI: https://doi.org/10.2307/2529548
Weinberg, M., & Williams, C. (1974). Male homosexuals: Their problems and adaptations. New York: Penguin.
Wohlwill, J. (1973). The Study of Behavioral Development. New York: Academic Press.