Predicting Research Productivity in International Evaluation Journals across Countries
Main Article Content
Abstract
Background: Progress in evaluation research depends on the continuous generation of scholarly knowledge and its dissemination in the community. One way of disseminating findings is to publish in scientific journals and researchers, institutions, and even whole countries are assessed by their output in these journals. Particularly with regard to countries, there is an uneven distribution of research productivity in evaluation journals. A viable model for predicting countries’ research output in international evaluation journals, however, has not yet been developed.
Purpose: The purpose of this study was to develop and test a model for the prediction of countries’ research output in international evaluation journals by predictors from the research, economic, and social/political system.
Setting: NA
Intervention: NA
Research Design: A cross-sectional design was used for predicting research output in evaluation journals across countries.
Data Collection and Analysis: Our sample consists of 65 countries that made contributions to ten international peer-reviewed evaluation journals. We collected data for the period from 2009 to 2013 and predicted the number of authorships across countries by using boosted regression trees, a machine learning procedure.
Findings: Our model provided accurate predictions of countries’ research output. Research productivity in the social sciences had the strongest effect, followed by economic prosperity, control of corruption, and age of evaluation society. The model was generalizable to another period of time with only marginal loss in predictive accuracy.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Basu, A. (2010). Does a country's scientific 'productivity' depend critically on the number of country journals indexed? Scientometrics, 82, 507-516. https://doi.org/10.1007/s11192-010-0186-8 DOI: https://doi.org/10.1007/s11192-010-0186-8
Canagarajah, A. S. (2002). A geopolitics of academic writing. Pittsburgh, PA: University of Pittsburgh Press. https://doi.org/10.2307/j.ctt5hjn6c DOI: https://doi.org/10.2307/j.ctt5hjn6c
Christie, C. A., & Fleischer, D. N. (2010). Insight into evaluation practice: A content analysis of designs and methods used in evaluation studies published in North American evaluation-focused journals. American Journal of Evaluation, 31, 326-346. https://doi.org/10.1177/1098214010369170 DOI: https://doi.org/10.1177/1098214010369170
Coryn, C. L. S., Ozeki, S., Wilson, L. N., Greenman II, G. D., Schröter, D. C., Hobson, K. A., Azzam, T., & Vo, A. T. (2016). Does research on evaluation matter? Findings from a survey of American Evaluation Association members and prominent evaluation theorists and scholars. American Journal of Evaluation, 37, 159-173. https://doi.org/10.1177/1098214015611245 DOI: https://doi.org/10.1177/1098214015611245
Diaz-Puente, J. M., Cazorla, A., & Dorrego, A. (2007). Crossing national, continental, and linguistic boundaries: Toward a worldwide evaluation research community in journals of evaluation. American Journal of Evaluation, 28, 399-415. https://doi.org/10.1177/1098214007308413 DOI: https://doi.org/10.1177/1098214007308413
Elith, J., Leathwick, J. R., & Hastie, T. (2008). A working guide to boosted regression trees. Journal of Animal Ecology, 77, 802-813. https://doi.org/10.1111/j.1365-2656.2008.01390.x DOI: https://doi.org/10.1111/j.1365-2656.2008.01390.x
Flowerdew, J. (1999). Writing for scholarly publication in English: The case of Hong Kong. Journal of Second Language Writing, 8, 123-145. https://doi.org/10.1016/S1060-3743(99)80125-8 DOI: https://doi.org/10.1016/S1060-3743(99)80125-8
Friedman, J. H. (2001). Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29, 1189-1232. https://doi.org/10.1214/aos/1013203451 DOI: https://doi.org/10.1214/aos/1013203451
Friedman, J. H. (2002). Stochastic gradient boosting. Computational Statistics & Data Analysis, 38, 367-378. https://doi.org/10.1016/S0167-9473(01)00065-2 DOI: https://doi.org/10.1016/S0167-9473(01)00065-2
Friedman, J. H., & Meulman, J. J. (2003). Multiple additive regression trees with application in epidemiology. Statistics in Medicine, 22, 1365-1381. https://doi.org/10.1002/sim.1501 DOI: https://doi.org/10.1002/sim.1501
Furubo, J.-E., & Sandahl, R. (2002). A diffusion perspective on global developments in evaluation. In J.-E. Furubo, R. C. Rist, & R. Sandahl (Eds.), International atlas of evaluation (pp. 1-23). New Brunswick, NJ: Transaction Publishers.
Gutiérrez, J., & López-Nieva, P. (2001). Are international journals of human geography really international? Progress in Human Geography, 25, 53-69. https://doi.org/10.1191/030913201666823316 DOI: https://doi.org/10.1191/030913201666823316
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). New York, NY: Springer New York. DOI: https://doi.org/10.1007/978-0-387-84858-7
Heberger, A. E., Christie, C. A., & Alkin, M. C. (2010). A bibliometric analysis of the academic influences of and on evaluation theorists' published works. American Journal of Evaluation, 31, 24-44. https://doi.org/10.1177/1098214009354120 DOI: https://doi.org/10.1177/1098214009354120
Husted, B. W. (1999). Wealth, culture, and corruption. Journal of International Business Studies, 30, 339-359. https://doi.org/10.1057/palgrave.jibs.8490073 DOI: https://doi.org/10.1057/palgrave.jibs.8490073
Jacob, S., Speer, S., & Furubo, J.-E. (2015). The institutionalization of evaluation matters: Updating the International Atlas of Evaluation 10 years later. Evaluation, 21, 6-31. https://doi.org/10.1177/1356389014564248 DOI: https://doi.org/10.1177/1356389014564248
Kaufmann, D., Kraay, A., & Mastruzzi, M. (2011). The worldwide governance indicators: Methodology and analytical issues. Hague Journal on the Rule of Law, 3, 220-246. https://doi.org/10.1017/S1876404511200046 DOI: https://doi.org/10.1017/S1876404511200046
Lane, P. R. (2011). Innovation and financial globalization. In J. Y. Lin & B. Pleskovic (Eds.), Annual World Bank conference on development economics - 2010: Lessons from East Asia and the global financial crisis (pp. 309-332). Washington D.C.: The World Bank.
Leeuw, F. L., & Furubo, J.-E. (2008). Evaluation systems: What are they and why study them? Evaluation, 14, 157-169. https://doi.org/10.1177/1356389007087537 DOI: https://doi.org/10.1177/1356389007087537
Love, A., & Russon, C. (2000). Building a worldwide evaluation community: Past, present, and future. Evaluation and Program Planning, 23, 449-459. https://doi.org/10.1016/S0149-7189(00)00035-5 DOI: https://doi.org/10.1016/S0149-7189(00)00035-5
Markiewicz, A. (2008). The political context of evaluation: What does this mean for independence and objectivity? Evaluation Journal of Australasia, 8, 35. https://doi.org/10.1177/1035719X0800800205 DOI: https://doi.org/10.1177/1035719X0800800205
Meo, S. A., Al Masri, Abeer A., Usmani, A. M., Memon, A. N., Zaidi, S. Z., & Preis, T. (2013). Correction: Impact of GDP, spending on R&D, number of universities and scientific journals on research publications among Asian countries. PLoS ONE, 8. https://doi.org/10.1371/annotation/3a739c2a-d5f2-4d6f-9e0d-890d5a54c33d DOI: https://doi.org/10.1371/annotation/3a739c2a-d5f2-4d6f-9e0d-890d5a54c33d
Nielsen, S. B., & Winther, D. M. (2014). A Nordic evaluation tradition? A look at the peer-reviewed evaluation literature. Evaluation, 20, 311-331. https://doi.org/10.1177/1356389014540703 DOI: https://doi.org/10.1177/1356389014540703
Origgi, G., & Ramello, G. B. (2015). Current dynamics of scholarly publishing. Evaluation Review, 39, 3-18. https://doi.org/10.1177/0193841X15572017 DOI: https://doi.org/10.1177/0193841X15572017
Ramsden, P. (1994). Describing and explaining research productivity. Higher Education, 28, 207-226. https://doi.org/10.1007/BF01383729 DOI: https://doi.org/10.1007/BF01383729
Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A systematic approach (7th ed.). Thousand Oaks, CA: Sage.
Schonlau, M. (2005). Boosted regression (boosting): An introductory tutorial and a Stata plugin. The Stata Journal, 5, 330-354. https://doi.org/10.1177/1536867X0500500304 DOI: https://doi.org/10.1177/1536867X0500500304
Short, J. R., Boniche, A., Kim, Y., & Li, P. L. (2001). Cultural globalization, global English, and geography journals. The Professional Geographer, 53, 1-11. https://doi.org/10.1111/0033-0124.00265 DOI: https://doi.org/10.1111/0033-0124.00265
Tanzi, V., & Schuknecht, L. (2000). Public spending in the 20th century: A global perspective. Cambridge, UK: Cambridge University Press. https://doi.org/10.1017/CBO9780511625800 DOI: https://doi.org/10.1017/CBO9780511625800
Vinluan, L. R. (2012). Research productivity in education and psychology in the Philippines and comparison with ASEAN countries. Scientometrics, 91, 277-294. https://doi.org/10.1007/s11192-011-0496-5 DOI: https://doi.org/10.1007/s11192-011-0496-5