Rubrics: A Method for Surfacing Values and Improving the Credibility of Evaluation
Main Article Content
Abstract
Background: The challenges of valuing in evaluation have been the subject of much debate; on what basis do we make judgments about performance, quality, and effectiveness? And according to whom? (Julnes, 2012b). There are many ways identified in the literature for carrying out assisted valuation (Julnes, 2012c). One way of assisting the valuation process is the use of evaluative rubrics.
This practice-based article unpacks the learnings of a group of evaluators who have used evaluative rubrics to grapple with this challenge. Compared to their previous practice, evaluative rubrics have allowed them to surface and deal with values in a more transparent way. In their experience when evaluators and evaluation stakeholders get clearer about values, evaluative judgments become more credible and warrantable.
Purpose: Share practical lessons learned from working with rubrics.
Setting: Aotearoa (New Zealand).
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: They have found that while evaluative rubrics look beguilingly simple they are hard to do well. However, when done well, evaluative rubrics can substantially increase the use and credibility of evaluation.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Davidson, E. J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks: Sage Publications. https://doi.org/10.4135/9781452230115 DOI: https://doi.org/10.4135/9781452230115
Davidson, E. J., Wehipeihana, N., & McKegg, K. (2011, September 1). The rubric revolution. Paper presented at the Australasian Evaluation Society Conference. Sydney, Australia.
Fournier, D. M. (1995). Establishing evaluative conclusions: A distinction between general and working logic. In D. M. Fournier (Ed.), Reasoning in Evaluation: Inferential Links and Leaps. New Directions for Evaluation, (58), 15-32. https://doi.org/10.1002/ev.1017 DOI: https://doi.org/10.1002/ev.1017
Gough, D. (2007). Weight of evidence: a framework for the appraisal of the quality and relevance of evidence. Research Papers in Education, 22, 213-228. https://doi.org/10.1080/02671520701296189 DOI: https://doi.org/10.1080/02671520701296189
House, E.R., & Howe, K. (1999). Values in evaluation and social research. Thousand Oaks: Sage. https://doi.org/10.4135/9781452243252 DOI: https://doi.org/10.4135/9781452243252
Hubbard, A. (2012, December 1). Of a scientific persuasion. Dominion Post. Retrieved from http://www.stuff.co.nz/dominion-post/capital-life/8023559/Of-a-scientific-persuasion
Julnes, G. (2012a). Developing Policies to Support Valuing in the Public Interest. In G. Julnes (Ed.), Promoting valuation in the public interest: Informing policies for judging value in evaluation. New Directions for Evaluation, 133, 109-129. https://doi.org/10.1002/ev.20012 DOI: https://doi.org/10.1002/ev.20012
Julnes, G. (2012b). Editor's notes. New Directions for Evaluation, 133, 1-2. https://doi.org/10.1002/ev.20001 DOI: https://doi.org/10.1002/ev.20001
Julnes, G. (2012c). Managing valuation. In G. Julnes (Ed.), Promoting valuation in the public interest: Informing policies for judging value in evaluation. New Directions for Evaluation, 133, 3-15. https://doi.org/10.1002/ev.20002 DOI: https://doi.org/10.1002/ev.20002
King, J. (2010). We all rely on good evaluation. Retrieved from Julian King & Associates Limited: http://www.julianking.co.nz/blog/good-evaluation/
McKegg, K., & Oakden, J. (2009). Characteristics of good peer support. Wellington: Wellink Trust.
Mertens, D. (2009). Transformative research and evaluation. New York: Guilford Press.
Ministry of Education. (2012, May 22). Measuring and reporting progress. Retrieved from Ministry of Education website.
New Zealand Qualifications Authority. (2009). Policy and guidelines for the conduct of external evaluation and review. Wellington, New Zealand: New Zealand Qualifications Authority.
Oakden, J. (2013). Evaluation rubrics: How to ensure transparent and clear assessment that respects diverse lines of evidence. Melbourne: Better Evaluation.
Oakden, J., & McKegg, K. (2011, August). Waste minimisation act implementation: Evaluation of stakeholder perspectives. Retrieved from Ministry for the Environment.
Oakden, J., & Wehipeihana, N. (2009). FTP Induction programme review (unpublished report). Wellington: Judy Oakden Consultancy.
Patton, M. Q. (2012). Contextual pragmatics of valuing. In G. Julnes (Ed.), Promoting valuation in the public interest: Informing policies for judging value in evaluation. New Directions for Evaluation, (133), 97-108. https://doi.org/10.1002/ev.20011 DOI: https://doi.org/10.1002/ev.20011
Pipi, K., Kennedy, V., Paipa, K., Akroyd, S. & King, J. (2012). Evaluation of the Te Puni Kōkiri investment in the Māori Wardens Project 2007-2010. Prepared for Te Puni Kōkiri. Auckland: FEM (2006) Ltd.
Popham, W. J. (2011). The role of rubrics in testing and teaching Mastering Assessment: A self-service system for educators, Pamphlet 13 (Mastering Assessment Series). Boston: Pearson.
Scriven, M. (1991). Evaluation thesaurus. Newbury Park, CA: Sage.
Scriven, M. (1995). The logic of evaluation and evaluation practice. In D. M. Fournier (Ed.), Reasoning in evaluation: Inferential links and leaps. New Directors for Evaluation, (68), 49-70. https://doi.org/10.1002/ev.1019 DOI: https://doi.org/10.1002/ev.1019
Scriven, M. (2012). The logic of valuing. In G. Julnes (Ed.). Promoting valuation in the public interest: Informing policies for judging value in evaluation. New Directions for Evaluation, (133), 17-28. DOI: https://doi.org/10.1002/ev.20003
Wehipeihana, N., King, J., Spee, K., Paipa, K., & Smith, R. (2010). Evaluation of He Ara Tika. Wellington, New Zealand: Ministry of Education.