The "Usability" of Evaluation Reports: A Precursor to Evaluation Use in Government Organizations
Main Article Content
Abstract
Background: According to the Treasury Board of Canada’s Policy on Evaluation (2009), evaluations produced by federal government departments must contribute to decision-making at an organizational level (mainly summative) as well as a program level (mainly formative). Previous research shows that although the formative objectives of evaluation are generally reached, the use of evaluation for broader, budgetary management is limited. However, little research has been conducted thus far on this issue.
Purpose: This study investigates the extent to which program evaluation is used in the Canadian federal government for budgetary management purposes.
Setting: This paper outlines the results obtained following the first component of a two-pronged research strategy focusing on evaluation use in Canadian federal government organizations.
Intervention: N/A
Research Design: Two federal agencies were recruited to participate in organizational case studies aiming to identify the factors that facilitate the use of evaluation for budgetary reallocation exercises.
Data Collection and Analysis: This report presents the findings from a detailed analysis of evaluation reports published by both agencies between 2010-2013. The data were collected from public evaluation reports and analyzed using NVivo.
Findings: The preliminary findings of the study show that instrumental use has occurred or can be expected to occur, based on the types of recommendations outlined in the reports reviewed and on the responses to the evaluations produced by program managers.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
Funding data
-
Social Sciences and Humanities Research Council
Grant numbers #430-2014-01091
References
Alkin, M. C., and Taut, S.M. (2002). Unbundling evaluation use. Studies in Educational Evaluation, 29, 1-12. https://doi.org/10.1016/S0191-491X(03)90001-0 DOI: https://doi.org/10.1016/S0191-491X(03)90001-0
Amo, C., and Cousins, J.B. (2007). Going through the process: An examination of the operationalization of process use in empirical research on evaluation. In J.B. Cousins (Ed.), Process use in theory, research and practice. New Directions for Evaluation (p. 5-26). San Francisco: Jossey-Bass. https://doi.org/10.1002/ev.240 DOI: https://doi.org/10.1002/ev.240
Bourgeois, I., Toews, E., Whynot, J., and Lamarche, M. K., (2013). Measuring organizational evaluation capacity in the Canadian federal government. Canadian Journal of Program Evaluation, 28 (2), 1-19. https://doi.org/10.3138/cjpe.28.001 DOI: https://doi.org/10.3138/cjpe.28.001
Cousins, J.B., & Leithwood, K.A. (1993). Enhancing knowledge utilization as a strategy for school improvement. Knowledge: Creation, Diffusion, Utilization, 14, 305-333. https://doi.org/10.1177/107554709301400303 DOI: https://doi.org/10.1177/107554709301400303
Dumaine, F., (2012). When one must go: The Canadian experience with strategic review and judging program value. In G. Julnes (Ed.), Promoting valuation in the public interest: Informing policies for judging value in evaluation. New Directions for Evaluation, 133, 65-75. https://doi.org/10.1002/ev.20007 DOI: https://doi.org/10.1002/ev.20007
Fleischer, D., and Christie, C.A. (2009). Evaluation use: A survey of U.S. American Evaluation Association members. American Journal of Evaluation, 30, 158-175. https://doi.org/10.1177/1098214008331009 DOI: https://doi.org/10.1177/1098214008331009
Johnson, K., Greenseid, L.O., Toal, S.A., King, J.A., Lawrenz, F., and Bolkov, B. (2009). Research on evaluation use: A review of the empirical literature from 1986 to 2005. American Journal of Evaluation, 30 (3), 377-410. https://doi.org/10.1177/1098214009341660 DOI: https://doi.org/10.1177/1098214009341660
Johnson, R. B. (1998). Toward a theoretical model of evaluation utilization. Evaluation and Program Planning, 21, 93-110. https://doi.org/10.1016/S0149-7189(97)00048-7 DOI: https://doi.org/10.1016/S0149-7189(97)00048-7
Leviton, L.C. (2003). Evaluation use: Advances, challenges and applications. American Journal of Evaluation, 24 (4), 525-535. https://doi.org/10.1177/109821400302400410 DOI: https://doi.org/10.1016/j.ameval.2003.10.003
McDavid, J. C., and Huse, I. (2012). Legislator uses of public performance reports: Findings from a five year, study. American Journal of Evaluation, 33 (1), 7-25. https://doi.org/10.1177/1098214011405311 DOI: https://doi.org/10.1177/1098214011405311
Government of Canada, Treasury Board Secretariat (2009). Policy on Evaluation, retrieved on May 3 2015, from: http://www.tbs-sct.gc.ca/pol/doc-fra.aspx?section=text&id=15024
Office of the Auditor General (2013). Spring Report of the Auditor General of Canada, Chapter 1 -Status Report on Evaluating the Effectiveness of Programs. Consulted online on April 26th, 2015 at: http://www.oag-bvg.gc.ca/internet/English/parl_oag_201304_01_e_38186.html#hd5m
Patton, M.Q. (1997). Utilization-focused Evaluation: The New Century Text. Thousand Oaks, CA: Sage Publications.