Evaluation in Our New Normal Environment: Navigating the Challenges with Data Collection
Main Article Content
Abstract
Background: Data collection is a critical component of all evaluations. However, it often presents a number of challenges under the best of circumstances. For instance, the evaluation budget and time frame both have implications for the quality and type of data that is collected. Additionally, adherence to high quality international ethical best practices is necessary when collecting data for any purpose, methodological rigor is important for ensuring the credibility of the evaluation, improving access to important documents and stakeholders, as well as decreasing excessive evaluation anxiety on the part of critical stakeholders, when possible, is vital. These challenges have now been considerably exacerbated by the COVID-19 global health pandemic which has changed our world in fundamental ways. In what is now considered as our new normal environment, evaluators will need to make profound changes to the manner in which they plan and undertake data collection.
Objectives: This paper examines the many and varied challenges that will be encountered with data collection in our new normal environment. This new normal has had an impact on evaluation practices in all countries, developed and developing, and has significantly amplified existing challenges in countries with limited evaluation culture, budgets, technological coverage, access, and connectivity. It makes an important contribution to the literature since data collection has historically and traditionally been conducted using primarily face-to-face field work and through the freedom of movement of people to undertake this task.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Desk review was utilized for the preparation of this paper.
Findings: Evaluators need to be extremely flexible, innovative, and amendable to different approaches to data collections as our new normal environment will likely be with us for a while. This pandemic has thrown everyone a very painful curveball and introduced significant new work-related challenges for a myriad of work types and work environments. Innovation and the willingness to learn new methods have become an important necessity to help with learning, accountability, transparency. The COVID-19 pandemic has highlighted the plight of the most vulnerable and evidence-based data is the only means to assist this group. Evaluators must rise to the challenge, devise new ways to collect data that is credible and useful, and continue to promote the importance and benefits of the field of evaluation. As such, evaluators have an important role to play in the global economic recovery efforts.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Bamberger, M., Rugh, J., Church, M., & Fort, L. (2004). Shoestring evaluation: Designing impact evaluations under budget, time, and data constraints. American Journal of Evaluation, 25(1), 5-37. https://doi.org/10.1177/109821400402500102 DOI: https://doi.org/10.1177/109821400402500102
Bamberger, M., Rugh, J., & Mabry, L. (2006). RealWorld evaluation: Working under budget, time, data, and political constraints. Thousand Oaks, CA: Sage Publications, Inc.
Bamberger, M. & Segone, M. (2011). How to design and manage equity-focused evaluations. UNICEF. https://evalpartners.org/sites/default/files/EWP5_Equity_focused_evaluations.pdf
Bethune, Z., & Korinek, A. (2020). COVID-19 infection externalities: Trading off lives vs. livelihoods. Working Pager 27009. Cambridge, MA; National Bureau of Economic Research. https://www.nber.org/papers/w27009 https://doi.org/10.3386/w27009 DOI: https://doi.org/10.3386/w27009
Canuto, O. (2020). The impact of coronavirus on the global economy (Policy Brief PB20-58). Policy Center for the New South. https://www.policycenter.ma/publications/impact-coronavirus-global-economy
Chelsky, J., & Kelly, L. (2020). Bowling in the dark: Monitoring and evaluation during COVID-19 (Coronavirus). World Bank Group. https://ieg.worldbankgroup.org/blog/mande-covid19
Clements, P., Chianca, T., & Sasaki, R. (2008). Reducing world poverty by improving evaluation of development aid. American Journal of Evaluation, 29(2), 195-214. https://doi.org/10.1177/1098214008318657 DOI: https://doi.org/10.1177/1098214008318657
Dayton, K., & Bolden, M. (2020). Continuing monitoring and evaluation efforts during the COVID-19 Pandemic: Q&A with i-APS. https://www.agrilinks.org/post/continuing-monitoring-and-evaluation-efforts-during-covid-19-pandemic-qa-i-aps
DeSilva, P. (2011, April). Practical challenges faced by evaluation professionals in undertaking consultancy assignments in evaluation. Paper presented at the International Development Evaluation Association Global Conference, Amman, Jordan. www.ideas-global.org/.../De-Silva-6.7-Challenges-faced-by-Evaluation-P
Donaldson, S., Gooler, L., & Scriven, M. (2002). Strategies for managing evaluation anxiety: Toward a psychology of program evaluation. American Journal of Evaluation, 23(3):261-273. https://doi.org/10.1177/109821400202300303 DOI: https://doi.org/10.1177/109821400202300303
Fisher, M. & Bubola, E. (2020). As coronavirus deepens inequality, inequality worsens its spread. New York Times. https://www.nytimes.com/2020/03/15/world/europe/coronavirus-inequality.html
Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2004). Program evaluation: Alternative approaches and practical guidelines. (3rd ed.). Boston, MA: Allyn and Bacon.
Gupta, A. (2020). Why some women call this recession a 'shecession'. New York Times. https://www.nytimes.com/2020/05/09/us/unemployment-coronavirus-women.html
Independent Expert Advisory Group. (2014). A World that counts: Mobilizing the data revolution for sustainable development. http://www.undatarevolution.org/wpcontent/uploads/2014/11/A-World-That-Counts.pdf
International Labour Office. (2020). ILO Monitor: COVID-19 and the world of work: Second edition updates estimates and analysis. https://www.ilo.org/wcmsp5/groups/public/---dgreports/---dcomm/documents/briefingnote/wcms_740877.pdf
Kelly, M. (2020). Economist: Social costs of COVID-19 outweigh individual costs. https://news.virginia.edu/content/economist-societal-costs-covid-19-outweigh-individual-costs
Kusek, J. Z. & Rist, R. C. (2004). Ten steps to a results-based monitoring and evaluation system. Washington, DC: The World Bank. https://doi.org/10.1596/0-8213-5823-5 DOI: https://doi.org/10.1596/0-8213-5823-5
Morris, M. (2015). Research on evaluation ethics: Reflections and an agenda. New Directions for Evaluation, 148, 31-42. https://doi.org/10.1002/ev.20155 DOI: https://doi.org/10.1002/ev.20155
Patton, M. Q. (2020). Evaluation implications of the coronavirus emergency. Social Science Space. https://socialsciencespace.com/2020/03/evaluation-implications-of-the-coronavirus-emergency/
Persaud, N. (2020). Data collection in our new normal, Caribbean EvalChat, 4(2).
Persaud, N. (in press). Strengthening evaluation culture in the English Speaking Commonwealth Caribbean: A guide for evaluation practitioners and decision-makers in the public, private, and NGO sectors. Kingston, Jamaica: Arawak Publications.
Persaud, N. (2016). General Challenges with conducting evaluations. Paper presented at a collaborative conference between the University of the West Indies, The Caribbean Development Bank, and Carleton University, Bridgetown, Barbados.
Persaud, N. & Dagher, R. (in press). The role of monitoring and evaluation in the United Nations 2030 sustainable development agenda. Palgrave McMillian Publishing.
Persaud, N., & Dagher, R. (2020). Evaluations in the English speaking Commonwealth Caribbean region: Lessons from the field. American Journal of Evaluation, 41(2), 255-276. https://doi.org/10.1177/1098214019866260 DOI: https://doi.org/10.1177/1098214019866260
Persaud, N., Devonish, D., & Persaud, I. (2019). Nuts and bolts of research methodology. From conceptualization to write-up. Kingston, Jamaica: Ian Randle Publishers Ltd.
Organization for Economic Cooperation and Development. (2020a). Coronavirus (COVID-19): SME policy responses. http://www.oecd.org/coronavirus/policy-responses/coronavirus-covid-19-sme-policy-responses-04440101/
Organization for Economic Cooperation and Development. (2020b). OECD policy responses to coronavirus (COVID-19): Women at the core of the fight against COVID-19 crisis. https://www.oecd.org/coronavirus/policy-responses/women-at-the-core-of-the-fight-against-covid-19-crisis-553a8269/
Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A systematic approach. (7th ed.). Thousand Oaks, CA: Sage Publications.
Royce, D., Thyer, B., Padgett, D. & Logan, T. (2001). Program evaluation: An introduction. (3rd ed.). Toronto: Wadsworth.
Stephens, K. K. et al. (2020). Collective sensemaking around COVID-19: Experiences, concerns and agendas for our rapidly changing organizational lives. Management Communication Quarterly, 34(3), 426-457. https://doi.org/10.1177/0893318920934890 DOI: https://doi.org/10.1177/0893318920934890
United Nations. (2019). The sustainable development goals report 2019. NY, USA: United Nations Publications. ISBN 978-92-1-101403-7
United Nations. (2020a). World economic situation and prospects as of mid-2020. https://www.un.org/development/desa/dpad/publication/world-economic-situation-and-prospects-as-of-mid-2020/
United Nations. (2020b). Brief#2: Putting the UN framework for socio-economic response to COVID-19 into action: Insights. United Nations Development Programme. https://www.undp.org/content/undp/en/home/coronavirus/socio-economic-impact-of-covid-19.html
World Bank. (2020). Global economic prospects. Washington, DC; World Bank Group. https://www.worldbank.org/en/publication/global-economic-prospects