AI for Evaluators: Opportunities and Risks
Article Sidebar
Main Article Content
Abstract
Background: The emergence of widely available and applicable artificial intelligence (AI) raises ethical, practical, and professional concerns for professional evaluators. The authors explore potential answers to emerging questions as to how evaluators can engage AI in an effective and responsible way.
Purpose: Advance the conversation around AI technology and its integration into professional evaluation practice.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: Authors explore two main use cases for AI: namely, proposal writing and evaluation design drafting. We also discuss four challenges for evaluators engaging with AI: The proliferation of the digital environment with excess output, market disruption and the emergence of new roles, the so-called "alignment problem", and the challenge to evaluators' to use their agency. Moving forward, the authors recommend evaluators familiarize themselves with AI technology, use it transparently, think critically about the effects of AI on their work, and use perspective when considering the potential ramifications of this new tool.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Brants, T., Popat, A. C., Xu, P., Och, F. J., & Dean, J. (2007). Large language models in machine translation. Google, Inc. http://research.google/pubs/pub33278.pdf
Bruce, K. (2023, March 5). ChatGPT and evaluation: 3 key takeaways [Post]. LinkedIn. https://www.linkedin.com/pulse/chatgpt-evaluation-3-key-takeaways-kerry-bruce/
Cantor, M. (2023, April 7). This economist won every bet he made on the future. Then he tested ChatGPT. The Guardian. https://www.theguardian.com/technology/2023/apr/06/chatgpt-ai-bryan-caplan-interview
Castelvecchi, D. (2016). Can we open the black box of AI? Nature News, 538(7623), 20–23. https://doi.org/10.1038/538020a DOI: https://doi.org/10.1038/538020a
Durán, J. M., & Jongsma, K. R. (2021). Who is afraid of black box algorithms? On the epistemological and ethical basis of trust in medical AI. Journal of Medical Ethics, 47(5), 329–335. https://doi.org/10.1136/medethics-2020-106820 DOI: https://doi.org/10.1136/medethics-2020-106820
Gabriel, I. (2020) Artificial intelligence, values, and alignment. Minds and Machines, 30, 411–437. https://doi.org/10.1007/s11023-020-09539-2 DOI: https://doi.org/10.1007/s11023-020-09539-2
High-Level Expert Group on Artificial Intelligence. (2019). A definition of AI: Main capabilities and disciplines. European Commission. https://ec.europa.eu/newsroom/dae/document.cfm?doc_id=60651
Kan, M. (2023, March 2). ChatGPT is coming to an app near you: OpenAI launches API for its chatbot. PC Magazine. https://www.pcmag.com/news/chatgpt-is-coming-to-an-app-near-you-openai-launches-api-for-its-chatbot
Marcus, G. (2022, December 1). How come GPT can seem so brilliant one moment and so breathtakingly dumb the next? The Road to AI We Can Trust. https://garymarcus.substack.com/p/how-come-gpt-can-seem-so-brilliant?
Microsoft. (2023, March 16). Introducing Microsoft 365 CopilotA whole new way to work. Microsoft 365 Blog. https://www.microsoft.com/en-us/microsoft-365/blog/2023/03/16/introducing-microsoft-365-copilot-a-whole-new-way-to-work/
OpenAI. (2022a, September 28). DALL·E now available without waitlist. OpenAI Blog. https://openai.com/blog/dall-e-now-available-without-waitlist/
OpenAI. (2022b, November 30). Introducing ChatGPT. OpenAI Blog. https://openai.com/blog/chatgpt/
Patton, M. Q. (2011). Essentials of utilization-focused evaluation. SAGE Publications.
Pethokoukis, J. (2016, June 6). What the story of ATMs and bank tellers reveals about the ‘rise of the robots’ and jobs. AEIdeas. https://www.aei.org/economics/what-atms-bank-tellers-rise-robots-and-jobs/
Scriven, M. (2013). The foundation and future of evaluation. In S. I. Donaldson (Ed.), The future of evaluation in society: A tribute to Michael Scriven (pp. 11–44). Information Age Publishing. DOI: https://doi.org/10.1108/978-1-62396-453-520251003
Simon, N. (2023, April 15). ChatGPT in evaluationAn opportunity for greater creativity? University World News. https://www.universityworldnews.com/post.php?story=20230412111133714
Walker, L. (2023, March 28). Belgian man dies of suicide following exchanges with chatbot. The Brussels Times. https://www.brusselstimes.com/430098/belgian-man-commits-suicide-following-exchanges-with-chatgpt/