The Elements to Enhance the Successful Start and Completion of Program and Policy Evaluations: The Injury & Violence Prevention (IVP) Program & Policy Evaluation Institute The Injury & Violence Prevention (IVP) Program & Policy Evaluation Institute
Main Article Content
Abstract
Background: Public health practitioners, including injury and violence prevention (IVP) professionals, are responsible for implementing evaluations, but often lack formal evaluation training. Impacts of many practitioner-focused evaluation trainings—particularly their ability to help participants successfully start and complete evaluations—are unknown.
Objectives: We assessed the impact of the Injury and Violence Prevention (IVP) Program & Policy Evaluation Institute (“Evaluation Institute”), a team-based, multidisciplinary, and practitioner-focused evaluation training designed to teach state IVP practitioners and their cross-sector partners how to evaluate program and policy interventions.
Design: Semi-structured interviews were conducted with members of 13 evaluation teams across eight states at least one year after training participation (24 participants in total). Document reviews were conducted to triangulate, supplement, and contextualize reported improvements to policies, programs, and practices.
Intervention: Teams of practitioners applied for and participated in the Evaluation Institute, a five-month evaluation training initiative that included a set of online training modules, an in-person workshop, and technical support from evaluation consultants.
Main Outcome Measure(s): The successful start and/or completion of a program or policy evaluation focused on an IVP intervention.
Results: Of the 13 teams studied, a total of 12 teams (92%) reported starting or completing an evaluation. Four teams (31%) reported fully completing their evaluations; eight teams (61%) reported partially completing their evaluations. Teams identified common facilitators and barriers that impacted their ability to start and complete their evaluations. Nearly half of the 13 teams (46%) – whether or not they completed their evaluation – reported at least one common improvement made to a program or policy as a result of engaging in an evaluative process.
Conclusion: Practitioner-focused evaluation trainings are essential to build critical evaluation skills among public health professionals and their multidisciplinary partners. The process of evaluating an intervention—even if the evaluation is not completed—has substantial value and can drive improvements to public health interventions. The Evaluation Institute can serve as a model for training public health practitioners and their partners to successfully plan, start, complete, and utilize evaluations to improve programs and policies.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Adams, J., & Dickinson, P. (2010). Evaluation Training to Build Capability in the Community and Public Health Workforce. American Journal of Evaluation, 31(3), 421-433. https://doi.org/10.1177/1098214010366586 DOI: https://doi.org/10.1177/1098214010366586
Bryan, R. L., Kreuter, M. W., & Brownson, R. C. (2008). Integrating Adult Learning Principles Into Training for Public Health Practice. Health Promotion Practice, 10(4), 557-563. https://doi.org/10.1177/1524839907308117 DOI: https://doi.org/10.1177/1524839907308117
Campbell, M., Fitzpatrick, R., Haines, A., Kinmonth, A. L., Sandercock, P., Spiegelhalter, D., & Tyrer, P. (2000). Framework for design and evaluation of complex interventions to improve health. BMJ, 321(7262), 694. https://doi.org/10.1136/bmj.321.7262.694 DOI: https://doi.org/10.1136/bmj.321.7262.694
Centers for Disease Control and Prevention. (2013). Step by Step – Evaluating Violence and Injury Prevention Policies. Retrieved from https://www.cdc.gov/injury/about/evaluation.html
Centers for Disease Control and Prevention. (2014). Using Evaluation to Inform CDC's Policy Process. Retrieved from Atlanta, GA: https://www.cdc.gov/policy/analysis/process/docs/UsingEvaluationtoInformCDCsPolicyProcess.pdf
Centers for Disease Control and Prevention. (2020). 10 Essential Public Health Services. Retrieved from https://www.cdc.gov/publichealthgateway/publichealthservices/essentialhealthservices.html
Chelimsky, E. (1987). Reports on Topic Areas What Have We Learned About the Politics of Program Evaluation? Evaluation Practice, 8(1), 5-21. https://doi.org/10.1177/109821408700800101 DOI: https://doi.org/10.1177/109821408700800101
Cousins, J. B., & Leithwood, K. A. (1986). Current Empirical Research on Evaluation Utilization. Review of Educational Research, 56(3), 331-364. https://doi.org/10.3102/00346543056003331 DOI: https://doi.org/10.3102/00346543056003331
Davies, R., & MacKay, K. (2014). Evaluator Training:Content and Topic Valuation in University Evaluation Courses. American Journal of Evaluation, 35(3), 419-429. https://doi.org/10.1177/1098214013520066 DOI: https://doi.org/10.1177/1098214013520066
Denford, S., Abraham, C., Callaghan, M., Aighton, P., De Vocht, F., & Arris, S. (2017). A review of grey and academic literature of evaluation guidance relevant to public health interventions. BMC Health Services Research, 17(1), 643. https://doi.org/10.1186/s12913-017-2588-2 DOI: https://doi.org/10.1186/s12913-017-2588-2
Denford, S., Lakshman, R., Callaghan, M., & Abraham, C. (2018). Improving public health evaluation: a qualitative investigation of practitioners' needs. BMC Public Health, 18(1), 190. https://doi.org/10.1186/s12889-018-5075-8 DOI: https://doi.org/10.1186/s12889-018-5075-8
Fierro, L. A., & Christie, C. A. (2011). Understanding Evaluation Training in Schools and Programs of Public Health. American Journal of Evaluation, 32(3), 448-468. https://doi.org/10.1177/1098214010393721 DOI: https://doi.org/10.1177/1098214010393721
Gebbie, K. M., Rosenstock, L., & Hernandez, L. M. (2003). Who Will Keep the Public Healthy? Educating Public Health Professionals for the 21st Century: National Academy Press.
Gibbs, D. A., Hawkins, S. R., Clinton-Sherrod, A. M., & Noonan, R. K. (2009). Empowering Programs With Evaluation Technical Assistance. Health Promotion Practice, 10(1_suppl), 38S-44S. https://doi.org/10.1177/1524839908316517 DOI: https://doi.org/10.1177/1524839908316517
Greene, J. C. (1987). Stakeholder participation in evaluation design: Is it worth the effort? Evaluation and Program Planning, 10(4), 379-394. https://doi.org/10.1016/0149-7189(87)90010-3 DOI: https://doi.org/10.1016/0149-7189(87)90010-3
Greene, J. G. (1988). Stakeholder Participation and Utilization in Program Evaluation. Evaluation Review, 12(2), 91-116. https://doi.org/10.1177/0193841x8801200201 DOI: https://doi.org/10.1177/0193841X8801200201
Hobson, K. A., Coryn, C. L. S., Fierro, L. A., & Sherwood-Laughlin, C. M. (2019). Instruction of Evaluation Competencies in Council on Education for Public Health (CEPH)-Accredited Master of Public Health (MPH) Degree Programs. American Journal of Evaluation, 40(4), 590-606. https://doi.org/10.1177/1098214019845510 DOI: https://doi.org/10.1177/1098214019845510
Kelly, C. M., LaRose, J., & Scharff, D. P. (2013). A Method for Building Evaluation Competency Among Community-Based Organizations. Health Promotion Practice, 15(3), 431-437. https://doi.org/10.1177/1524839913496427 DOI: https://doi.org/10.1177/1524839913496427
LaVelle, J. M. (2020). Educating Evaluators 1976–2017: An Expanded Analysis of University-Based Evaluation Education Programs. American Journal of Evaluation, 41(4), 494-509. https://doi.org/10.1177/1098214019860914 DOI: https://doi.org/10.1177/1098214019860914
LaVelle, J. M., & Donaldson, S. I. (2010). University-Based Evaluation Training Programs in the United States 1980—2008: An Empirical Examination. American Journal of Evaluation, 31(1), 9-23. https://doi.org/10.1177/1098214009356022 DOI: https://doi.org/10.1177/1098214009356022
Mayberry, R. M., Daniels, P., Akintobi, T. H., Yancey, E. M., Berry, J., & Clark, N. (2008). Community-based Organizations' Capacity to Plan, Implement, and Evaluate Success. Journal of Community Health, 33(5), 285-292. https://doi.org/10.1007/s10900-008-9102-z DOI: https://doi.org/10.1007/s10900-008-9102-z
Milstein, B., & Wetterhall, S. F. (1999). Framework for program evaluation in public health. Morbidity and Mortality Weekly Report, 48 (RR-11).
Napp, D., Gibbs, D., Jolly, D., Westover, B., & Uhl, G. (2002). Evaluation Barriers and Facilitators Among Community-Based HIV Prevention Programs. AIDS Education and Prevention, 14(3_supplement), 38-48. https://doi.org/10.1521/aeap.14.4.38.23884 DOI: https://doi.org/10.1521/aeap.14.4.38.23884
Pell Institute for the Study of Opportunity in Higher Education, the Institute for Higher Education Policy, & Pathways to College Network. The Evaluation Toolkit. Retrieved from http://toolkit.pellinstitute.org/
Safe States Alliance. (2013). Building Safer States: Core Components of State Public Health Injury and Violence Prevention Programs. Retrieved from Atlanta, GA, : https://cdn.ymaws.com/www.safestates.org/resource/resmgr/files/2013_buildingsaferstates.pdf
Scriven, M. (2007). The Logic of Evaluation. OSSA Conference Archive, 138. https://scholar.uwindsor.ca/ossaarchive/OSSA7/papersandcommentaries/138/
Thompson, N., & McClintock, H. (2000). Demonstrating Your Program's Worth: A Primer on Evaluation for Programs to Prevent Unintentional Injury (Revised).
Trevisan, M. S. (2004). Practical Training in Evaluation: A Review of the Literature. American Journal of Evaluation, 25(2), 255-272. https://doi.org/10.1177/109821400402500212 DOI: https://doi.org/10.1016/S1098-2140(04)00023-2
Vaughan, R. (2004). Evaluation and Public Health. American Journal of Public Health, 94(3), 360. https://doi.org/10.2105/ajph.94.3.360 DOI: https://doi.org/10.2105/AJPH.94.3.360
World Health Organization. (2020). The Public Health Approach. Retrieved from https://www.who.int/violenceprevention/approach/public_health/en/