A Practical Framework and Model for Promoting Cost-Inclusive Evaluations
Main Article Content
Abstract
Background: Cost studies are an important component of any serious professional evaluation. Regardless of whether an evaluation is conducted in the public or private sector, decision-makers want answers to two fundamental questions: (1) Is the program, project, or investment worthwhile? and (2) Can funds be used in a better way? The answers to both questions rely on the use of cost analysis techniques. However, according to the literature, most evaluations do not include any type of cost study (see Christie & Fleischer, 2010) and when studies are included, very few are of high quality and rigor (see Madsen, Eddleston, Hansen, & Konradsen, 2017). This may in part be as a result of the relatively limited emphasis placed on this important topic in the evaluation-specific literature, coupled with the reality that many evaluators lack expertise to conduct cost analysis studies.
Purpose: Given the limited use of cost studies in professional evaluations to assess actual program merit/worth, this paper presents a practical framework/model to help evaluators understand fundamental issues that must be considered when thinking about some form of cost-inclusive evaluation.
Setting: Not Applicable.
Intervention: Not Applicable.
Research Design: External desk research was used to ascertain the extent to which cost analysis is discussed in the evaluation literature and the reasons highlighted for underuse. This review provided the foundation for developing the practical framework/model presented in this paper for promoting cost-inclusive evaluation.
Data Collection and Analysis: Not Applicable.
Findings: Cost analysis, which can considerably enrich professional evaluations, is currently quite underused in the evaluation profession. Notable progress has been made over the last 15 years with government agencies around the globe taking the lead to promote cost analysis. However, most independent evaluators are still failing to measure, report, and analyze costs because many do not understand the fundamentals of cost studies. This paper promotes cost-inclusive evaluation by providing evaluators with a framework/model to help them understand important issues in cost analysis.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Alkin, M. C. & Christie, C. A. (2004). An evaluation theory tree. In M. E. Alkin (Ed.). Evaluation roots: Tracing theorists' views and influences (12-65). Thousand Oaks, CA: Sage. https://doi.org/10.4135/9781412984157.n2 DOI: https://doi.org/10.4135/9781412984157.n2
Chawla, K. (1990). Social cost-benefit analysis. New Delhi, India: Mittal Publications.
Cellini, S. R. & Kee, J. E. (2015). Cost-effectiveness and cost-benefit analysis. In K. E. Newcomer, H. P. Hatry, & J. S. Wholey (Eds.). Handbook of practical program evaluation (4th ed.) (636672). Hoboken, NJ: John Wiley & Sons.
Chelimsky, E. (1997). The political environment of evaluation and what it means for the development of the field. In E. Chelimsky & W. R. Shadish (Eds.), Evaluation for the 21st century: A handbook (53-68). Thousand Oaks, CA: Sage. https://doi.org/10.4135/9781483348896.n3 DOI: https://doi.org/10.4135/9781483348896.n3
Christie, C.A., & Fleischer, D. N. (2010). Insight into evaluation practice: A content analysis of designs and methods used in evaluations studies published in North American evaluation-focused journals. American Journal of Evaluation, 31(3), 326-346. https://doi.org/10.1177/1098214010369170 DOI: https://doi.org/10.1177/1098214010369170
Copeland, C. W. (2013). Economic analysis and independent regulatory agencies. Retrieved from https://www.acus.gov/sites/default/files/documents/Copeland%20Final%20BCA%20Report%204-30-13.pdf
Davidson, J. E. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage. https://doi.org/10.4135/9781452230115 DOI: https://doi.org/10.4135/9781452230115
Department of Finance and Administration, Australia (2006). Handbook of cost-benefit analysis. Retrieved from www.finance.gov.au/publications/.../Handbook_of_CBanalysis.pdf
Drummond, M. F., Sculpher, M.J., Claxton, K., Stoddart, G. L., & Torrance, G. W. (2015). Methods of economic evaluation of health care programmes (4th ed.). Oxford, UK: Oxford University Press.
European Commission. (2014). European commission guide to cost-benefit analysis of investment projects: Economic appraisal tool for cohesion policy 2014-2020. Retrieved from http://ec.europa.eu/regional_policy/sources/docgener/studies/pdf/cba_guide.pdf
Fals-Stewart, W., Yates, B.T., & Klostermann, K. (2005). Assessing the costs, benefits, costbenefit ratio, and cost-effectiveness of marital and family treatments: Why we should and how we can. Journal of Family Psychology, 19, 2839. https://doi.org/10.1037/0893-3200.19.1.28 DOI: https://doi.org/10.1037/0893-3200.19.1.28
Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative approaches and practical guidelines (4th ed.). Upper Saddle Road, NJ: Pearson Education.
French. M. T. (2003). Drug abuse treatment cost analysis program (DATCAP): User's manual (8th ed.). Miami, FL: University of Miami. Retrieved from http://www.datcap.com/downloads/DATCAPUser_Manual.pdf
Gilpin, A. (1995). Environmental impact assessment: Cutting edge for the twenty-first century. Cambridge, UK: Cambridge University Press. https://doi.org/10.1017/CBO9781139166539 DOI: https://doi.org/10.1017/CBO9781139166539
Gramlich, E. M. (1981). Benefit-cost analysis of government programs. Englewood Cliffs, NJ: Prentice Hall.
Herman, P. M., Avery, D. J., Schemp, C. S., & Walsh, M. E. (2009). Are cost-inclusive evaluations worth the effort? Evaluation and Program Planning, 32(1), 55-61. https://doi.org/10.1016/j.evalprogplan.2008.08.008 DOI: https://doi.org/10.1016/j.evalprogplan.2008.08.008
HM Treasury. (2011). The green book: Appraisal and evaluation in central government. Retrieved from www.hmtreasury.gov.uk/d/green_book_complete.pdf
Kee, J. E. (1994). Benefit-cost analysis in program evaluation. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.). Handbook of practical program evaluation (2nd ed.) (pp. 456-488). San Francisco, CA: Jossey-Bass.
Kusek, J. Z. & Rist, R. C. (2004). Ten steps to a results-based monitoring and evaluation system. Washington, DC: The World Bank. https://doi.org/10.1596/0-8213-5823-5 DOI: https://doi.org/10.1596/0-8213-5823-5
Levin, H. M. (2005). Cost-benefit analysis. In S. Mathison (Ed.), Encyclopedia of evaluation (pp. 86). Thousand Oaks, CA: Sage.
Levin, H. M., & McEwan, J. P. (2001). Cost-effectiveness analysis: Methods and applications (2nd ed.). Thousand Oaks, CA: Sage.
Madsen, L. B., Eddleston, M., Hansen, K. S., & Konradsen, F. (2017). Quality assessment of economic evaluations of suicide and self-harm interventions: A systematic review. Crisis, 1- 14. Advance online publication. http://dx.doi.org/10.1027/0227-5910/a000476 https://doi.org/10.1027/0227-5910/a000476 DOI: https://doi.org/10.1027/0227-5910/a000476
Mohr, B. L. (1995). Impact analysis for program evaluation. Thousand Oaks, CA: Sage.
National Archives (2017). Executive order 12291- Federal regulation. Retrieved September 15, 2017 from http://www.archives.gov/federalregister/codification/executiveorder/12291.html
New Zealand Treasury. (2005). Cost benefit analysis premier. Wellington, New Zealand: The Treasury.
New Zealand Treasury. (2015). Guide to social cost benefit analysis. Retrieved from http://www.treasury.govt.nz/publications/guidance/planning/costbenefitanalysis/guide/
Nicholson, C. & Branker Greene, S. (2017). Tips for evaluation on a shoestring budget. Caribbean EvalChat 3(1), 2-3.
Office of Information and Regulatory Affairs (n.d.). Agency checklist: Regulatory Impact Analysis. Retrieved from https://obamawhitehouse.archives.gov/sites/default/files/omb/Inforeg/regpol/RIA_Checklist.pdf
Office of Management and Budget. (1992). Guidelines and discount rates for benefit-cost analysis of federal programs: Circular No. A-94 revised. Retrieved from http://www.whitehouse.gov/omb/circulars_default
Office of Management and Budget. (2003). Circular A-4. Retrieved from https://www.whitehouse.gov/omb/circulars_a004_a-4
Office of Management and Budget. (2011). Regulatory Impact Analysis: A Premier. Retrieved from https://www.regulationwriters.com/downloads/Circular-A-4- RIA-Primer.pdf
Office of Management and Budget. (2016). United States office of management and budget handbook. International Business Publications, Washington: DC.
Persaud, N. (2005). Is cost analysis underutilized in decision making? Journal of MultiDisciplinary Evaluation, 2(2), 81-82. https://doi.org/10.56645/jmde.v2i2.122 DOI: https://doi.org/10.56645/jmde.v2i2.122
Persaud, N. (2007). Conceptual and practical analysis of costs and benefits in evaluation: Developing a cost analysis tool for practical program evaluation USA: ProQuest Information and Learning Company. Retrieved from http://scholarworks.wmich.edu/dissertations/906
Persaud, N. (2009). Cost analysis. In C. Wankel (Ed.), Encyclopedia of business in today's world, Vol. 1 (pp. 415). Thousand Oaks, CA: Sage.
Posavac, E. J., & Carey, R. G. (2003). Program evaluation: Methods and case studies (6th ed.). Upper Saddle Road, NJ: Prentice Hall.
Rossi, H. P., Lipsey, W. M., & Freeman, E. H. (2004). Evaluation: A systematic approach (7th ed). Thousand Oaks, CA: Sage.
Royce, D., Thyer, B. A., Padgett, D. K., & Logan, T. K. (2001). Program evaluation: An introduction. Belmont, CA: Thomson Learning Inc.
Scriven, M. (1991). Evaluation thesaurus (4th ed.). Thousand Oaks, CA: Sage.
Scriven, M. (2007). Key evaluation checklist. Retrieved from https://wmich.edu/sites/default/files/attachments/u350/2014/key%20evaluation%20checklist.pdf
Scriven, M. (2015). Key evaluation checklist. Retrieved from http://michaelscriven.info/
Shaffer, M. (2010). Multiple account benefit-cost analysis: A practical guide for the systematic evaluation of project and policy alternative. Toronto, Canada: University of Toronto Press. https://doi.org/10.3138/9781442686632 DOI: https://doi.org/10.3138/9781442686632
The Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed). Thousand Oaks, CA: Sage Publications.
The White House. (2011). Executive order 13563: Improving regulation and regulatory review. Retrieved from https://www.whitehouse.gov/the-pressoffice/2011/01/18/executive-order-13563-improving-regulation-and-regulatory-review
Treasury Board of Canada Secretariat (2007). Canadian cost-benefit analysis guide: Regulatory proposals. Retrieved September 15, 2016 from http://www.tbssct.gc.ca/reports-rapports/index-eng.asp
Weiss, C. H. (1998). Evaluation. Upper Saddle River, NJ: Prentice Hall.
Wholey, J. S. (2003). Improving performance and accountability: Responding to emerging management challenges. In S. I. Donaldson and M. Scriven (Eds.). Evaluating social programs and problems (pp. 43-61). Mahwah, NJ. Lawrence Erlbaum Associates.
Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (2004). Handbook of practical program evaluation (2nd ed.). San Francisco, CA: JosseyBass.
World Health Organization (2003). Introduction to drug utilization research. Oslo, Norway: World Health Organization. Retrieved from http://apps.who.int/medicinedocs/en/d/Js48 76e/
Yates, B. T. (1996). Analyzing costs, procedures, processes, and outcomes in human services. Thousand Oaks, CA: Sage Publications. https://doi.org/10.4135/9781412983358 DOI: https://doi.org/10.4135/9781412983358
Yates, B. T. (1999). Measuring and improving cost, cost-effectiveness, and cost-benefit for substance abuse treatment programs. Rockville, MD: National Institute on Drug Abuse. NIH Publication Number 99-4518, also: http://www.nida.nih.gov/IMPCOST/IMPCOSTIndex.html https://doi.org/10.1037/e597582007-001 DOI: https://doi.org/10.1037/e597582007-001
Yates, B. T. (2009). Cost-inclusive evaluation: A banquet of approaches for including costs, benefits, and cost-effectiveness and cost benefit analyses in your next evaluation. Evaluation and Program Planning, 32: 52-54. https://doi.org/10.1016/j.evalprogplan.2008.08.007 DOI: https://doi.org/10.1016/j.evalprogplan.2008.08.007
Yates, B. T., & Marra, M. (20171). Social return on investment (SROI): Problems, solutions...and is SROI a good investment? Evaluation and Program Planning, 64: 136-144. https://doi.org/10.1016/j.evalprogplan.2016.11.009 DOI: https://doi.org/10.1016/j.evalprogplan.2016.11.009
Yates, B. T., and Marra, M. (20172). Introduction: Social return on investment (SROI). Evaluation and Program Planning, 64: 95-97. https://doi.org/10.1016/j.evalprogplan.2016.10.013 DOI: https://doi.org/10.1016/j.evalprogplan.2016.10.013