Method for Using Rubric Ratings on Fishbone Diagrams to Compare Case Studies
Main Article Content
Abstract
Background: In multi-case study program evaluations, the large amount of qualitative data that are generated from interviews can be difficult to utilize. This is particularly so when inference must be made as to why some cases succeed and some fail.
Purpose: This paper shows a method for comparing multiple evaluation sites by using a rubric to define ratings for relevant factors, and an Ishikawa fishbone diagram as a model to show relationships among those factors. We show how this technique identified reasons for differences in outcomes among the sites.
Setting: The evaluation setting was a large-scale safety innovation in the U.S. railroad industry. Four cases were considered—two passenger railroads and two freight railroads.
Intervention: The Confidential Close Call Reporting System (C3RS) program allowed railroad workers to confidentially submit “close calls” which were reviewed by a team made up of labor, management, and the Federal Railroad administration to determine ways to improve safety.
Research design: Multiple comparative case study, Ishikawa root and contributing cause modeling, evaluative rubric scoring, and data visualization techniques.
Data collection & analysis: Interview data were collected from four pilot railroad sites, each of which participated in a five-year test of C3RS. Testing periods overlapped, with the entire evaluation lasting about 12 years.
Findings: The method of using Ishikawa fishbone diagrams with ratings from an evaluative rubric was an effective method to summarize, analyze, and present large quantities of qualitative data. The approach succeeded in explaining degrees of success and failure across the sites. The sponsor and industry stakeholders were able to understand the analysis and the findings, and to develop deep insight into how to promote successful implementation.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
Funding data
References
Brinkerhoff, R. O. (2005). The success case method: A strategic evaluation approach to increasing the value and effect of training. Advances in Developing Human Resources, 7(1), 86-101. https://doi.org/10.1177/1523422304272172 DOI: https://doi.org/10.1177/1523422304272172
Creswell, J. W. (2015). A concise introduction to mixed methods research. Thousand Oaks, CA: Sage.
Davidson, J. E. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage. https://doi.org/10.4135/9781452230115 DOI: https://doi.org/10.4135/9781452230115
Evergreen, S. D. H. (2014). Presenting data effectively: Communicating your findings for maximum impact. Thousand Oaks, CA: Sage.
Ishikawa, K. (1982). Guide to quality control. Asian Productivity Organization.
Jones, N. D., Azzam, T., Linnel Wanzer, D., Skousen, D., Knight, C., & Sabarre, N. (2019) Enhancing the effectiveness of logic models. American Journal of Evaluation, online first. https://doi.org/10.1177/1098214018824417 DOI: https://doi.org/10.1177/1098214018824417
Malamed, C. (2011). Visual language for designers: Principles for creating graphics that people understand. Rockport Publishers.
Medina, L. Acosta-Perex, E., Velez, C., Martinex, G., Rivera, M., Sardinas, L., Pattatucci, A. (2015) Training and capacity building evaluation: Maximizing resources and results with the Success Case Method. Evaluation and Program Planning, 52, 126-132. https://doi.org/10.1016/j.evalprogplan.2015.03.008 DOI: https://doi.org/10.1016/j.evalprogplan.2015.03.008
Patton, M. Q. (2015). Qualitative research and evaluation methods (4th ed.). Thousand Oaks, CA: Sage.
Patton, M. Q. (1987). How to use qualitative methods in evaluation. Thousand Oaks, CA: Sage.
Ranney, J. M., Davey, M., Morell, J., Zuschlag, M., & Kidda, S. (2019) Confidential Close Call Reporting System (C3RS) lessons learned evaluation-Final report. Federal Railroad Administration. DOT/FRA/ORD-19/01.
Yin, R. K. (2013). Case study research: Design and methods (Applied Social Research Methods). (5th ed.). Thousand Oaks, CA: Sage.