How should Program Evaluation Standards inform the use of cost-benefit analysis in evaluation?
Article Sidebar
Main Article Content
Abstract
Background: Cost-benefit analysis (CBA), like any other evaluation method, should be used in ways that uphold program evaluation standards and should be subjected to metaevaluation. In contrast to the broad remit of program evaluation standards, guidelines for economic evaluation focus mainly on technical aspects of evaluation quality, aimed at ensuring precision, accuracy, and reliability. Can CBA be conducted in adherence both to program evaluation standards and to its own methodological principles, or are there areas where expectations conflict?
Purpose: Assess the potential for CBA to be conducted in keeping with the Program Evaluation Standards (PES) of the Joint Committee on Standards for Educational Evaluation.
Setting: Analysis applies to any setting in which CBA is being considered as an evaluation method.
Intervention: N/A
Research Design: Methodological principles underpinning CBA were systematically assessed against the PES, to determine the extent to which CBA can be conducted in a manner aligned with these standards. CBA was rated according to whether it can follow each standard in principle, not the extent to which economists follow a given standard in practice.
Data Collection and Analysis: This assessment was undertaken from a theoretical perspective, through analysis of relevant literature. The ratings are evaluative; they represent the judgments of the author, made on the basis of explicit definitions.
Findings: Some ethical principles espoused in the PES are also required in CBA. On the other hand, some of the PES are not explicit requirements in CBA, though they could be applied by evaluators or economists when conducting a CBA. However, some PES logically cannot be met by CBA if it is used as a stand-alone method. All PES can theoretically be met when an evaluation combines CBA with other methods. In order to use CBA in adherence to PES, evaluators and economists must take an explicit interest in the effects of their analysis on people’s lives. This has significant implications for the way CBA should be used, including the nature and extent of stakeholder involvement, the potential use of CBA in conjunction with other methods, and decisions about when not to use CBA. As with any evaluation method, deliberation is necessary over whether, when, and how to use CBA.
Keywords: cost-benefit analysis, program evaluation standards, metaevaluation
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Adler, M.D., & Posner, E.A. (2006). New Foundations of Cost-Benefit Analysis. Cambridge, Mass: Harvard University Press. DOI: https://doi.org/10.2307/j.ctv1nzfgqt
Allan C., Kerr, S., Grimes, A. (2013). Value and Culture. Motu Working Paper 13-09. Wellington: Motu Economic and Public Policy Research. DOI: https://doi.org/10.29310/wp.2013.09
American Economic Association. (2018). AEA Code of Professional Conduct, adopted April 20, 2018. [web page]. Retrieved from: https://www.aeaweb.org/about-aea/code-of-conduct
American Evaluation Association. (2018). Guiding Principles for Evaluators. [web page]. Retrieved from: https://www.eval.org/About/Guiding-Principles
ANZEA & Superu. (2015). Evaluation standards for Aotearoa New Zealand. Wellington, NZ: Aotearoa New Zealand Evaluation Association and Social Policy Evaluation and Research Unit.
Argyrous, G. (2013). A review of government cost-benefit analysis guidelines. SSC/ANZSOG Occasional Paper. Canberra, Australia: Australia and New Zealand School of Government.
Australian Evaluation Society. (2013). Guidelines for the Ethical Conduct of Evaluations. Retrieved from: https://www.aes.asn.au/images/AES_Guidelines_web_v2.pdf?type=file
Backhouse, R.E. (2016, August). The origins of New Welfare Economics. Preliminary draft written to provoke discussion at a workshop in welfare economics, Hitotsubashi University, Japan.
Boston, J., & Gill, D. (Eds.). (2017). Social investment: A New Zealand policy experiment. Wellington, New Zealand: Bridget Williams Books. DOI: https://doi.org/10.7810/9781988533582
Chapple, S. (2017). Corked wine in a cracked bottle. In J. Boston & D. Gill (Eds.), Social investment: A New Zealand policy experiment (pp. 355- 379). Wellington, New Zealand: Bridget Williams Books. DOI: https://doi.org/10.7810/9781988533582_18
Coryn, C.L.S. & Stufflebeam, D.L. (2014). Evaluation Theory, Models, & Applications. San Francisco, CA: Jossey-Bass.
Creedy, J., & Passi, H. (2017). Public sector discount rates: a comparison of alternative approaches. Working Paper 17/02. Wellington, New Zealand: NZ Treasury.
Damart, S., Roy, B. (2009). The uses of cost-benefit analysis in public transportation decision-making in France. Transport Policy, 16, 200- 212. DOI: https://doi.org/10.1016/j.tranpol.2009.06.002
Davis, K.E., Frank, R.G. (1992). Integrating costs and outcomes. New Directions for Evaluation (54), 69-84. DOI: https://doi.org/10.1002/ev.1609
Deane, K.L., & Harré, N. (2016). Developing a thoughtful approach to evaluation: Values-driven guidelines for novice evaluators. Evaluation Matters—He Take Tō Te Aromatawai, 2, 53-78. DOI: https://doi.org/10.18296/em.0011
Destremau, K., & Wilson, P. (2017). Defining social investment, Kiwi-style. In J. Boston & D. Gill (Eds.), Social investment: A New Zealand policy experiment (pp. 32-79). Wellington, New Zealand: Bridget Williams Books. DOI: https://doi.org/10.7810/9781988533582_2
Drummond, M. F., Sculpher, M. J., Torrance, G. W., O’Brien, B. J., & Stoddard, G. L. (2005). Methods for the economic evaluation of health care programs. Oxford, England: Oxford University Press. DOI: https://doi.org/10.1093/oso/9780198529446.001.0001
Exec. Order No. 13563, 3 C.F.R. Page 3821 (2011).
Fournier, D. M. (1995). Establishing evaluative conclusions: A distinction between general and working logic. In D. M. Fournier (Ed.), Reasoning in Evaluation: Inferential Links and Leaps. New Directions for Evaluation, (58), 15-32. DOI: https://doi.org/10.1002/ev.1017
Frank, R. (2000). Why is cost-benefit analysis so controversial? In M.D. Adler & E.A. Posner (Eds.), Cost-Benefit Analysis: Legal, Economic, and Philosophical Perspectives (pp. 77-94). Chicago, IL: University of Chicago Press.
Fujiwara, D., & Campbell, R. (2011). Valuation Techniques for Social Cost-Benefit Analysis: Stated Preference, Revealed Preference and Subjective Well-Being Approaches – A Discussion of the Current Issues. Government of the United Kingdom. Retrieved from: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/209107/greenbook_valuationtechniques.pdf
Greene, J.C. (2005). The generative potential of mixed methods inquiry. Westminster Studies in Education, 28(2), 207-211. DOI: https://doi.org/10.1080/01406720500256293
HM Treasury. (2022). The Green Book: Central government guidance on appraisal and evaluation. Government of the United Kingdom. Retrieved from: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1063330/Green_Book_2022.pdf
House, E.R., Howe, K.R. (1999). Values in Evaluation and Social Research. Sage. DOI: https://doi.org/10.4135/9781452243252
Husereau, D., Drummond, M., Petrou, S., Carswell, C., Moher, D., Greenberg, D., Augustovski, F., Briggs, A.H., Mauskopf, J., and Loder, E. (2013). Consolidated Health Economic Evaluation Reporting Standards (CHEERS) Statement. Cambridge University Press. DOI: https://doi.org/10.1186/1741-7015-11-80
Julnes, G. (2012a). Editor’s Notes. In G. Julnes (Ed). Promoting Valuation in the Public Interest: Informing Policies for Judging Value in Evaluation. New Directions for Evaluation, 133, pp 1-2.
Julnes, G. (2012b). Managing valuation. In G. Julnes (Ed). Promoting Valuation in the Public Interest: Informing Policies for Judging Value in Evaluation. New Directions for Evaluation, 133, pp 3-15. DOI: https://doi.org/10.1002/ev.20002
Julnes, G. (2012c). Promoting valuation in the public interest. In G. Julnes (Ed). Promoting Valuation in the Public Interest: Informing Policies for Judging Value in Evaluation. New Directions for Evaluation, 133, pp 109-129. DOI: https://doi.org/10.1002/ev.20012
King, J. (2015). Use of cost-benefit analysis in evaluation. Letter to the editor. Evaluation Journal of Australasia, 15(3), 37-41. DOI: https://doi.org/10.1177/1035719X1501500305
King, J. (2017). Using Economic Methods Evaluatively. American Journal of Evaluation, 38(1), 101-113. DOI: https://doi.org/10.1177/1098214016641211
King, J. (2019). Evaluation and Value for Money: Development of an approach using explicit evaluative reasoning. (Doctoral dissertation). Melbourne, Australia: University of Melbourne.
Levin, H.M., McEwan, P.J. (2001). Cost-Effectiveness Analysis. 2nd Ed. Thousand Oaks: Sage.
Levy, H., & Sarnat, M. (1994). Capital Investment & Financial Decisions (5th Ed). Hertfordshire, United Kingdom: Prentice Hall.
MacLennan, S., Stead, I., Little, A. (2021). Wellbeing Guidance for Appraisal: Supplementary Green Book Guidance. HM Treasury, Government of the United Kingdom. Retrieved from: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1005388/Wellbeing_guidance_for_appraisal_-_supplementary_Green_Book_guidance.pdf
Mertens, D.M., Hessie-Biber, S. (2013). Mixed methods and credibility of evidence in evaluation. In D.M. Mertens & S. Hessie-Biber (Eds.), Mixed methods and credibility of evidence in evaluation. New Directions For Evaluation, 138, 5-13. DOI: https://doi.org/10.1002/ev.20053
Mulgan, R.G. (1984). Who should have how much say about what? Some Problems in Pluralist Democracy. Political Science, 36(2), 112-124. DOI: https://doi.org/10.1177/003231878403600202
Nicholls, J., Lawlor, E., Neitzert, E., Goodspeed, T. (2012). A Guide to Social Return on Investment. January 2012. Haddington, England: The SROI Network.
OECD DAC. (2012). DAC Guidelines and Reference Series. Quality Standards for Development Evaluation. Evaluation Network of the Development Assistance Committee (DAC) of the Organisation for Economic Co-operation and Development (OECD). Retrieved
Patel, M. (2013). African Evaluation Guidelines. African Evaluation Journal, 1(1), 1-5. DOI: https://doi.org/10.4102/aej.v1i1.51
Patton, M.Q. (2017). Evaluation Flash Cards: Embedding evaluative thinking in organizational culture. St Paul, MN: Otto Bremer Trust.
Pinkerton, S.D., Johnson-Masotti, A.P., Derse, A., Layde, P.M., (2002). Ethical issues in cost-effectiveness analysis. Evaluation and Program Planning 25, 71-83. DOI: https://doi.org/10.1016/S0149-7189(01)00050-7
Schwandt, T. (2015). Evaluation Foundations Revisited: Cultivating a Life of the Mind for Practice. Redwood City: Stanford University Press.
Scriven, M. (1980). The logic of evaluation. Thousand Oaks, CA: Sage.
Scriven, M. (1991). Evaluation thesaurus. Newbury Park, CA: Sage.
Scriven, M. (1994). The Final Synthesis. Evaluation Practice, 15(3), 367-382. DOI: https://doi.org/10.1016/0886-1633(94)90031-0
Scriven, M. (1995). The Logic of Evaluation and Evaluation Practice. New Directions For Evaluation, 68, 49-70. DOI: https://doi.org/10.1002/ev.1019
Scriven, M. (2012). The logic of valuing. In G. Julnes (Ed.), Promoting valuation in the public interest: Informing policies for judging value in evaluation. New directions for evaluation, 133, 17-28. DOI: https://doi.org/10.1002/ev.20003
Sinden, A., Kysar, D. A., & Driesen, D. M. (2009). Cost-benefit analysis: New foundations on shifting sand. Regulation & Governance, 3, 48 DOI: https://doi.org/10.1111/j.1748-5991.2009.01044.x