Evaluation Policy and Organizational Evaluation Capacity Building: Application of an Ecological Framework across Cultural Contexts
Main Article Content
Abstract
Background: Research on the role and effects of evaluation policy is limited. Some research on the policy’s role in enhancing organizational evaluation capacity (EC) is beginning to accrue but to date it has been limited largely to global Western evaluation contexts.
Purpose: We employed an ecological conceptual framework arising from our own empirical research to explore the interface between evaluation policy and EC in non-western contexts. We asked—To what extent does this framework resonate across these contexts? In the selected non-Western context, what are the salient variables moderating the relationship between policy and EC in the selected contexts? Are there differences across countries?
Setting: The present research is focused on perceptions about evaluation culture and experiences in two countries situated in the Middle East and North Africa (MENA) region, namely Turkey and Jordan.
Intervention: Not applicable.
Research design: We conducted focus groups within the respective countries with a combined total of 18 participants associated with country-level voluntary organizations for professional evaluation (VOPE). Participants worked in government, non-governmental aid agencies, universities and private sector organizations.
Data collection and analysis: We introduced the focus group participants to our ecological framework and then guided the conversation using semi-structured questions. Data were audio-recorded, transcribed and subsequently thematically analyzed using NVivo.
Findings: The ecological framework was found to resonate well but the findings were weighted heavily toward macro-level contextual variables. Even though important contextual and cultural differences between Turkey and Jordan were evident, leadership emerged as a significant meso-level moderating variable in both settings. The discussion of the results included implications for ongoing research.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
Funding data
References
Acree, J., & Chouinard, J. (2019). Exploring use and influence in Culturally Responsive Approaches to Evaluation: A review of the empirical literature. American Journal of Evaluation, 1-5. https://doi.org/10.1177/109821401987950
Al Hudib, H., Cousins, J. B., Oza, J., Lakshminayarana, U., & Bhat, V. D. (2016). A cross-cultural evaluation conversation in India: Benefits, challenges and lessons learned. Canadian Journal of Program Evaluation, 30(3), 329-343. https://doi.org/10.3138/cjpe.30.3.06 DOI: https://doi.org/10.3138/cjpe.30.3.06
Al Hudib, H., & Cousins, J. B. (2019, submitted). Understanding evaluation policy and organizational capacity for evaluation: An interview study. Under review for publication consideration.
Al Hudib, H., & Cousins, J. B. (2020, forthcoming). Evaluation policy and organizational evaluation capacity building In M. M. Mark & N. Hart (Eds.), Evaluation Policy, New Directions for Evaluation. Thousand Oaks: SAGE.
Alonso, J., & Glennie, J. (2016). What is development cooperation? 2016 Development Cooperation Forum Policy Briefs (1), 1-5.
Armytage, L. (2011). Evaluating aid: An adolescent domain of practice. Evaluation, 17(3), 261-276. https://doi.org/10.1177/1356389011410518 DOI: https://doi.org/10.1177/1356389011410518
Askew, C., Beverly, M., & Jay, L. (2012). Aligning collaborative and culturally responsive evaluation approaches. Evaluation and Program Planning, 35, 552-557. https://doi.org/10.1016/j.evalprogplan.2011.12.011 DOI: https://doi.org/10.1016/j.evalprogplan.2011.12.011
Bronfenbrenner, U. (1979). The ecology of human development: Experiments by nature and design. Cambridge, MA: Harvard University Press. https://doi.org/10.4159/9780674028845 DOI: https://doi.org/10.4159/9780674028845
Carden, F. (2013). Evaluation, not development evaluation. American Journal of Evaluation, 34(4), 576-579. https://doi.org/10.1177/1098214013495706 DOI: https://doi.org/10.1177/1098214013495706
Chouinard, J. (2016). Introduction: Decolonizing international development evaluation. Canadian Journal of Program Evaluation, 30(3), 237-247. https://doi.org/10.3138/cjpe.30.3.01 DOI: https://doi.org/10.3138/cjpe.30.3.01
Chouinard, J., & Cram, F. (2019). Situating culturally responsive approaches to evaluation: Empirical implications for theory and practice. Thousand Oaks: SAGE. https://doi.org/10.4135/9781506368559 DOI: https://doi.org/10.4135/9781506368559
Chouinard, J. A., & Cousins, J. B. (2009). A review and synthesis of current research on cross-cultural evaluation. American Journal of Evaluation, 30(4), 457-494. https://doi.org/10.1177/1098214009349865 DOI: https://doi.org/10.1177/1098214009349865
Chouinard, J., & Hopson, R. (2016). A critical exploration of International Development Evaluation. Canadian Journal of Program Evaluation, 30(3), 248-276. https://doi.org/10.3138/cjpe.30.3.02 DOI: https://doi.org/10.3138/cjpe.30.3.02
Chouinard, J., & Milley, P. (2016). Mapping the spatial dimensions of participatory practice: A discussion of context in evaluation. Evaluation and Program Planning, 54, 1-10. https://doi.org/10.1016/j.evalprogplan.2015.09.003 DOI: https://doi.org/10.1016/j.evalprogplan.2015.09.003
Christie, C., & Lemire, S. (2019). Why evaluation theory should be used to inform evaluation policy. American Journal of Evaluation, 40(4), 490-508. https://doi.org/10.1177/1098214018824045 DOI: https://doi.org/10.1177/1098214018824045
Coldwell, M. (2019). Reconsidering context: Six underlying features of context to improve learning from evaluation. Evaluation, 25(1), 99-117. https://doi.org/10.1177/1356389018803234 DOI: https://doi.org/10.1177/1356389018803234
Dabelstein, N., & Patton, M. Q. (2013). Lessons learned and the contributions of the Paris Declaration: Evaluation to evaluation theory and practice. Canadian Journal of Program Evaluation, 27(3), 173–200. DOI: https://doi.org/10.3138/cjpe.0027.009
Dillman, L., & Christie, C. (2017). Evaluation policy in a non-profit foundation: a case study exploration of the Robert Wood Johnson foundation. American Journal of Evaluation, 31(1), 60-79. https://doi.org/10.1177/1098214016642864 DOI: https://doi.org/10.1177/1098214016642864
Greaves, S. (2012). A primer of Middle Eastern leadership culture. Journal of Strategic Security, 5(4), 99-118. https://doi.org/10.5038/1944-0472.5.4.7 DOI: https://doi.org/10.5038/1944-0472.5.4.7
Fairholm, G.W. (1994). Leadership and the Culture of Trust. Praeger, Westport, CT. https://doi.org/10.5040/9798400677595 DOI: https://doi.org/10.5040/9798400677595
Fierro, L. (2019, November). Research on federal evaluation policies. Paper presented at the annual meeting of the American Evaluation Association, Minneapolis.
Frierson, H., Hood, S., Hughes, G., & Thomas, V. (2010). A guide to conducting culturally responsive evaluations. In J. Frechting (Ed.), The 2010 user-friendly handbook for project evaluation. Alexandria, VA: National Science Foundation. https://doi.org/10.1187/cbe.10
Hay, K. (2010). Evaluation field building in South Asia: Reflections, anecdotes, and questions. American Journal of Evaluation, 31(2), 222-231. https://doi.org/10.1177/1098214010366175 DOI: https://doi.org/10.1177/1098214010366175
Hood, S., Hopson, R., & Frierson, P. (Eds.). (2005). The role of culture and cultural context: A mandate for inclusion, the discovery of truth and understanding in evaluative theory and practice. Greenwich, CT: Information Age Publishing.
Hood, S., Hopson, R., & Kirkhart, K. (2015). Culturally responsive evaluation: Theory, practice and future implications. In K. Newcomer, H. P. Hatry & J. S. Wholey (Eds.), Handbook of Practical Program Evaluation (4th Ed.). (pp. 281-317). Hoboken, NJ: Wiley. https://doi.org/10.1002/9781119171386.ch12 DOI: https://doi.org/10.1002/9781119171386.ch12
Jackson, E. T. (2016). Paternalistic leadership: The missing link in cross-cultural leadership studies? International Journal of Cross Cultural Management, 16(1), 3-7. https://doi.org/10.1177/147059581663770 DOI: https://doi.org/10.1177/1470595816637701
Kinarsky, A., & Christie, C. (2019, November). Evaluation policy in philanthropies: What are they? Who has them? Why write them? Paper presented at the annual meeting of the American Evaluation Association, Minneapolis. https://doi.org/10.1002/ev.20372 DOI: https://doi.org/10.1002/ev.20372
Mansur, J., Sobral, F., & Goldszmidt, R. (2017). Shades of paternalistic leadership across cultures. Journal of World Business, 52(5), 702-713. https://doi.org/10.1016/j.jwb.2017.06.003 DOI: https://doi.org/10.1016/j.jwb.2017.06.003
Mayne, J. (2007). Challenges and lessons in implementing Results-Based Management. Evaluation, 13(1), 87-109. https://doi.org/10.1177/1356389007073683 DOI: https://doi.org/10.1177/1356389007073683
Mayne, J. (2009). Results management: Can results evidence gain a foothold in the public sector? In O. Reiper, F. Leeuw, & T. Ling (Eds.), The evidence book (pp. 117–149). New Brunswick, NJ: Transaction.
Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Thousand Oaks: Sage.
Patton, M. Q. (1985). Editor's Notes Culture and evaluation, New Directions for Program Evaluation, No 25 (pp. 1-2). San Francisco: Jossey-Bass. https://doi.org/10.1002/ev.1380 DOI: https://doi.org/10.1002/ev.1380
Patton, M. Q. (2002). Qualitative research & evaluation methods (3rd ed.). Thousand Oaks, CA: SAGE
Pawson, R. & Tilley, N. (1997). Realistic Evaluation. London: SAGE.
Pawson, R. (2013). The Science of Evaluation: A Realist Manifesto. London: SAGE. https://doi.org/10.4135/9781473913820 DOI: https://doi.org/10.4135/9781473913820
Pawson, R. (2016). The ersatz realism of critical realism: A reply to Porter. Evaluation, 22(1), 49-57. https://doi.org/10.1177/1356389015605206 DOI: https://doi.org/10.1177/1356389015605206
Raimondo, E. (2018). The power and dysfunctions of evaluation systems in international organizations. Evaluation, 24(1), 26-41. https://doi.org/10.1177/1356389017749068 DOI: https://doi.org/10.1177/1356389017749068
Rist, R. C. (2006). The "E" in monitoring and evaluation: Using evaluative knowledge to support a results based management system. In R. C. Rist & N. Stame (Eds.), From studies to streams: Managing evaluative systems (pp. 3–22). New Brunswick, NJ: Transaction. https://doi.org/10.4324/9780203791189-1 DOI: https://doi.org/10.4324/9780203791189-1
Sachs, J. D. (2012). From Millennium Development Goals to Sustainable Development Goals. The Lancet, 379 (9382), 2206–2211. https://doi.org/10.1016/S0140-6736(12)60685-0 DOI: https://doi.org/10.1016/S0140-6736(12)60685-0
Sarayrah, Y. K. (2004). Servant leadership in the Bedouin-Arab culture. Global Virtue Ethics Review, (5) 3, 50–58.
Theisohn, T. & Courtnadge, P. (2005). Moving beyond the "Munchhausen Approach". Honest brokering and independent monitoring in development partnerships, in Nord-Sud aktuell, 19, 246-258.
Trochim, W. M. K. (2009). Evaluation policy and evaluation practice. New Directions for Evaluation, No. 123, 13–32. https://doi.org/10.1002/ev.303 DOI: https://doi.org/10.1002/ev.303
Vo, A. T., & Christie, C. A. (2015). (Eds.). Advancing research on evaluation through the study of context. New Directions for Evaluation, No. 148, 43–55. https://doi.org/10.1002/ev.20156 DOI: https://doi.org/10.1002/ev.20156
Wilkins, P., & Boyle, R. (2011). Standards and qualities. In J. Lonsdale, P. Wilkins, & T. Ling (Eds.), Performance audit: Contributing to accountability in democratic government, 147–173. Cheltenham, UK: Edward Elgar. https://doi.org/10.4337/9780857931801.00014 DOI: https://doi.org/10.4337/9780857931801.00014