Comparison Group Identification for Difficult-to-Evaluate Populations: Lessons from Evaluating 1n10 LGBTQ Youth Support Services
Main Article Content
Abstract
The evaluation literature on Lesbian, Gay, Bisexual, Transgender, and Queer (LGBTQ) youth programs is largely absent of quantitative studies, likely in part due to the challenge of using experimental or quasi-experimental evaluation designs. This paper proposes the creative use of a national data set to overcome the problem of estimating a counterfactual for this population. In addition to discussing lessons from this approach, we describe the program and its impacts. Evidence suggests that the program under study—1n10, a local support group for LGBTQ youth in the Phoenix, AZ, metropolitan area—draws a relatively more disadvantaged group than the national average but that their levels of suicide ideation are lower. By addressing key threats to the internal validity of evaluation designs, we rule out plausible rival explanations for program impacts.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Aiken, L. S., West, S. G., Schwalm, D. E., Carroll, J., & Hsuing, S. (1998). Comparison of a randomized and two quasi- experimental designs in a single outcome evaluation: Efficacy of a university-level remedial writing program. Evaluation Review, 22(4), 207-244.
https://doi.org/10.1177/0193841X9802200203
Bamberger, M., Rugh, J., Church, M., & Fort, L. (2003). Shoestring evaluation: Designing impact evaluations under budget, time and data constraints. American Journal of Evaluation, 25(1), 5-37.
https://doi.org/10.1177/109821400402500102
Beals, J., Spicer, P., Mitchell, C., Novins, D., Manson, S., & the Al-SUPERPFP Team. (2003). Racial disparities in alcohol use: Comparison of two american indian reservation populations with national Data. American Journal of Public Health, 93 (10), 1683-1685.
https://doi.org/10.2105/AJPH.93.10.1683
Bell, A., & Weinberg, M. (1978). Homosexualities: A study of diversity among men and women. New York: Simon and Shuster.
Bloom, H., Michalopoulos, C., Hill, C.,& Ying, L. (2002). Can nonexperimental comparison group methods match the findings from a random assignment evaluation of mandatory welfare-to-work programs?, Working Paper, Manpower Demonstration Research Corporation.
Cochran, W. G., & Chambers, S. P. (1965). The planning of observational studies of human populations. Journal of the Royal Statistical Society. Series A, 128 (2), 234-266.
https://doi.org/10.2307/2344179
Cook, T. D., Shadish, W. R., & Wong, V. C. (2005). Within study comparisons of experiments and non-experiments: Can they help decide on evaluation policy? (pp. 1-46): French Economic Society Meeting on Program Evaluation.
D'Agostino, R. B., & Kwan, H. (1995). Measuring effectiveness: What to expect without a randomized control group. Medical Care, 33(Supplement), AS95-AS105.
D'Agostino, R. B., & Rubin, D. B. (2000). Estimating and using propensity scores with partially missing data. Journal of the American Statistical Association, 95(451), 749-759.
https://doi.org/10.1080/01621459.2000.10474263
D'Augelli, A., & Hershberger, S. (1993). Lesbian, gay, and bisexual youth in community settings: Personal challenges and mental health problems. American Journal of Community Psychology, 21, 421-448.
https://doi.org/10.1007/BF00942151
DeCrescenzo, T. (1992). The brave new world of gay and lesbian youth. In B. Berzon (Ed.), Positively Gay. Berkeley, CA: Celestial Arts.
DuRant, R. H., Krowchuk, D., & Sinal, S. (1998). Victimization, use of violence, and drug use at school among male adolescents who engage in same-sex sexual behavior Journal of Pediatrics, 133(1), 113-118.
https://doi.org/10.1016/S0022-3476(98)70189-1
Faulkner, A. H., & Cranston, K. (1998). Correlates of same-sex sexual behavior in a random sample of Massachusetts high school students. American Journal of Public Health, 88(2), 262-266.
https://doi.org/10.2105/AJPH.88.2.262
Garofalo, R., Wolf, R. C., Kessel, S., Palfrey, S. J., & DuRant, R. H. (1998). The association between health-risk behaviors and sexual orientation among a school-based sample of adolescents. Pediatrics, 101, 895-902.
https://doi.org/10.1542/peds.101.5.895
Grossman, A. H., & Kerner, M. S. (1998). Self Esteem and supportiveness as predictors of emotional distress in gay male and lesbian youth. Journal of Homosexuality, 35(2), 25-39
https://doi.org/10.1300/J082v35n02_02
Hammelman, T. (1993). Gay and lesbian youth: Contributing factors to serious attempts or considerations of suicide. Journal of Gay and Lesbian Psychotherapy, 2(1), 77-89.
https://doi.org/10.1300/J236v02n01_06
Herdt, G., & Boxer, A. (1993). Children of Horizons. Boston: Beacon Press.
Joyner, K., & Udry, J. R. (2000). You don't bring me anything but down: Adolescent romantic relationships and depression. Journal of Health and Social Behavior, 41(December), 369-391.
https://doi.org/10.2307/2676292
Kourany, R. (1987). Suicide among homosexual adolescents. Journal of Homosexuality, 13, 111- 117.
https://doi.org/10.1300/J082v13n04_07
Kruks, G. (1991). Gay and lesbian homeless street youth: Special issues and concerns. Journal of Adolescent Health, 12, 515-518.
https://doi.org/10.1016/0197-0070(91)90080-6
Lee, C. (2002). The impact of belonging to a high school gay/straight alliance. The High School Journal, Feb/Mar 2002.
https://doi.org/10.1353/hsj.2002.0005
LeVay, S., & Nonas, E. (1995). City of FriendsA A portrait of the gay and lesbian community in America. Cambridge, MA: The MIT Press.
Lock, J., & Steiner, H. (1998). Gay, Lesbian, and Bisexual Youth Risks for Emotional, Physical, and Social Problems: Results from a community-based survey. Journal of the American Academy of Child and Adolescent Psychiatry, 38(3), 297-304.
https://doi.org/10.1097/00004583-199903000-00017
Marsh, H. W. (1998). Simulation study of non- equivalent group-matching and regression- discontinuity designs: Evaluation of gifted and talented programs. Journal of Experimental Education, 66, 163-192.
https://doi.org/10.1080/00220979809601402
Nesmith, A., Burton, D., & Cosgrove, T. (1999). Gay, lesbian, and bisexual youth and young adults: Social support in their own words. Journal of Homosexuality, 37(1), 95-108.
https://doi.org/10.1300/J082v37n01_07
Noell, J., & Ochs, L. (2001). Relationship of Sexual Orientation to Substance Abuse, Suicidal Ideation, Suicide Attempts, and Other Factors in a Population of Homeless Adolescents. Journal of Adolescent Health, 29(1), 31-36.
https://doi.org/10.1016/S1054-139X(01)00205-1
Patton, M. C. (1982). Practical Evaluation. Newbury Park, CA: Sage.
Peck, L. R. (2003). Subgroup Analysis in Social Experiments: Measuring Program Impacts Based on Post-Treatment Choice. American Journal of Evaluation, 24(2), 157-187.
https://doi.org/10.1177/109821400302400203
Pfeiffer, S., & Strzelecki, B. (1990). Inpatient Psychiatric Treatment of Children and Adolescents: A Review of Outcome Studies. Journal of the American Academy of Child and Adolescent Psychiatry, 29, 847-853.
https://doi.org/10.1097/00004583-199011000-00001
Ramfedi, G., Farrow, J., & Deisher, R. (1991). Risk factors for attempted suicide in gay and bisexual yoth. Pediatrics, 87, 869-875.
https://doi.org/10.1542/peds.87.6.869
Ramfedi, G., French, S., Story, M., Resnick, M., & Blum, R. (1998). The relationship between suicide risk and sexual orientaion: Results of a population-based study. American Journal of Public Health, 88(1), 57-60.
https://doi.org/10.2105/AJPH.88.1.57
Rosenbaum, P. R., & Rubin, D. B. (1983). The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika, 70(1), 41-55.
https://doi.org/10.1093/biomet/70.1.41
Rosenbaum, P. R., & Rubin, D. B. (1984). Reducing Bias in Observational Studies Using Subclassification on the Propensity Score. Journal of the American Statistical Association, 79(387), 516-524.
https://doi.org/10.1080/01621459.1984.10478078
Rosenberg, M. (1979). Conceiving the self. New York: Basic Books.
Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A Systematic Approach. Thousand Oaks, CA: Sage.
Rubin, D. B., & Thomas, N. (1996). Matching Using Estimated Propensity Scores: Relating Theory to Practice. Biometrics, 52, 249-264.
https://doi.org/10.2307/2533160
Russell, S. T., & Consolacion, T. B. (2003). Adolescent Romance and Emotional Health in the United States: Beyond Binaries. Journal of Clinical and Adolescent Psychology 32(4), 499-508.
https://doi.org/10.1207/S15374424JCCP3204_2
Russell, S. T., Franz, B. T., & Driscoll, A. K. (2001). Same-sex Romantic Attraction and Experiences of Violence in Adolescence. American Journal of Public Health, 91(6), 903- 906.
https://doi.org/10.2105/AJPH.91.6.903
Russell, S. T., & Joyner, K. (2001). Adolescent Sexual Orientation and Suicide Risk:
https://doi.org/10.2105/AJPH.91.8.1276
Evidence from a national study. American Journal of Public Health, 91(8), 1276-1281.
Russell, S. T., & Seif, H. (202). Bisexual female adolescents: A critical review of past research, and results from a national study. Journal of Bisexuality, 2, 73-94.
https://doi.org/10.1300/J159v02n02_05
Russell, S. T., Seif, H., & Truong, N. L. (2001). School Outcomes of Sexual Minority Youth in the United States: Evidence from a national study. Journal of Adolescence, 24(1), 111-127.
https://doi.org/10.1006/jado.2000.0365
Saghir, M., & Robins, E. (1973). Male and female homosexuality: A comprehensive investigation. Baltimore: Williams and Wilkins.
Savin-Williams, R. (1990). Gay and lesbian youth: Expressions of identity. New York: Hemisphere Publishing Corp.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.
Shaw, R., Rosati, M., Salzman, P., Coles, C., & McGeary, C. (1997). Effects of adolescent ATOD behaviors and attitudes of a 5-year community partnership. Evaluation and Program Planning, 20 (3), 307-313.
https://doi.org/10.1016/S0149-7189(97)00010-4
Shifrin, F., & Solis, M. (1992). Chemical dependency in gay and lesbian youth. Journal of Chemical Dependency Treatment, 5(1), 67-76.
https://doi.org/10.1300/J034v05n01_06
Smith, H. L. (1997). Matching with Multiple Controls to Estimate Treatment Effects in Observational Studies. Soicological Methodology, 27, 325-353.
https://doi.org/10.1111/1467-9531.271030
Smith, J. A., & Todd, P. E. (2001). Reconciling Conflicting Evidence on the Performance of Propensity-Score Matching Methods. The American Economic Review, 91(2), 112-118.
https://doi.org/10.1257/aer.91.2.112
Ury, H. K. (1975). Efficiency of Case-Control Studies with Multiple Controls per Case: Continuos or Dichotomous Data. Biometrics, 31, 643-649.
https://doi.org/10.2307/2529548
Weinberg, M., & Williams, C. (1974). Male homosexuals: Their problems and adaptations. New York: Penguin.
Wohlwill, J. (1973). The Study of Behavioral Development. New York: Academic Press.