Peer Review of Abstracts Submitted to the Graduate Student and New Evaluators Topical Interest Group for the 2006 American Evaluation Association Conference
Main Article Content
Abstract
Peer review is an umbrella term that refers to a class of selection and oversight practices, including the familiar mechanisms of the review of proposals submitted for funding, of manuscripts for scholarly publications, and of personnel qualifications and portfolios for selection and promotion. Peer review has long been a cornerstone of modern scientific method premised on the assumption that those within a discipline are best suited to assess the work of others within that field. As such, it is also frequently employed to evaluate proposals submitted for professional meetings such as the annual conference of the American Evaluation Association (AEA). This paper presents a blind peer review method developed by AEA’s Graduate Student & New Evaluators (GS&NE) Topical Interest Group (TIG) in an effort to construct an impartial and reliable process in proposal selection. Implications for conference review processes, AEA, and the field of evaluation in general are discussed.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Aksnes, D. W. (2005). Citations and their use as indicators in science policy: Studies of validity and applicability issues with a particular focus on highly-cited papers. Unpublished doctoral dissertation, University of Twente, Netherlands.
Alkin, M. C., Daillak, R. H. (1979). A study of evaluation utilization. Educational Evaluation and Policy Analysis, 1(4), 41-49. https://doi.org/10.3102/01623737001004041 DOI: https://doi.org/10.3102/01623737001004041
Azzam, T. (2007). Evaluator contextual responsiveness. Unpublished doctoral dissertation, University of California, Los Angeles.
Barela, E. (2005, October). How school district evaluators make sense of their practice: A folk theory. Paper presented at the Joint Conference of the Canadian Evaluation Society and the American Evaluation Association, Toronto, Canada.
Barnett, E., Costantino, T., Hood, L., Jang, E. E., & Walker, K. C. (2004). Evaluation of the 2003 annual conference of the American Evaluation Association: Final report. Retrieved on May 10, 2007 from http://www.eval.org/training/eval history.asp
Bartholomay, T., Lilligren, L., Smith, J., Volkov, B., Williems, G., & King, J. A. (2001). AEA 2001 Conference evaluation: Final report. Retrieved on May 10, 2007 from http://www.eval.org/training/evalhistory.asp
Benos, D. J., Bashari, E., Chaves, J. M., Gaggar, A., Kapoor, N., LaFrance, M., et al. (2007). The ups and downs of peer review. Advances in Physiological Education, 31. 145-152. https://doi.org/10.1152/advan.00104.2006 DOI: https://doi.org/10.1152/advan.00104.2006
Burnham, J. C. (1992). How journal editors came to develop and critique peer review procedures. In H. F. Mayland & R. E. Sojka (Eds.), Research ethics, manuscript review and journal quality (pp. 55-62). Madison, WI: ACS Miscellaneous Publication. https://doi.org/10.2134/1992.researchethics.c5 DOI: https://doi.org/10.2134/1992.researchethics.c5
Bush, V. (1960). Science-The endless frontier: A report to the President on a program for postwar scientific research. Washington, DC: National Science Foundation. (Original work published in 1946).
Campanario, J. M. (1998). Peer review for journals as it stands today-Part 1. Science Communication, 19(3), 181-211. https://doi.org/10.1177/1075547098019003002 DOI: https://doi.org/10.1177/1075547098019003002
Ceci, S. J., & Peters, D. P. (1982). Peer review-a study of reliability. Change, 14, 44-48.
Chandler, M. (2001, November). How evaluators engage theory and philosophy in their practice. Paper presented at the annual conference of the American Evaluation Association, St. Louis, MO. https://doi.org/10.1080/00091383.1982.10569910 DOI: https://doi.org/10.1080/00091383.1982.10569910
Chelimsky, E. (1997). The political environment of evaluation and what it means for the development of the field. In E. Chelimsky & W. R. Shadish (Eds.), Evaluation for the 21stcentury: A handbook (pp. 53-68). Thousand Oaks, CA: Sage. https://doi.org/10.4135/9781483348896.n3 DOI: https://doi.org/10.4135/9781483348896.n3
Christie, C. A. (2003). What guides evaluation? A study of how evaluation practice maps onto evaluation theory. In C. A. Christie (Ed.), New directions for evaluation: The practice-theory relationship in evaluation (Vol.97, pp. 7-35). San Francisco, CA: Jossey-Bass. https://doi.org/10.1002/ev.72 DOI: https://doi.org/10.1002/ev.72
Christie, C. A. (2007). Reported influence of evaluation data on decision makers' actions. American Journal of Evaluation, 28(1), 8-25. https://doi.org/10.1177/1098214006298065 DOI: https://doi.org/10.1177/1098214006298065
Christie, C. A., & Masyn, K. E. (2007). Latent profiles of evaluators' self-reported practices. Manuscript submitted for publication.
Cole, S., Cole, J. R., & Simon, G. A. (1981). Chance and consensus in peer review. Science, 214, 881-886. https://doi.org/10.1126/science.7302566 DOI: https://doi.org/10.1126/science.7302566
Coryn, C. L. S. (2006). The use and abuse of citations as indicators of research quality. Journal of MultiDisciplinary Evaluation, 3(4), 115-120. https://doi.org/10.56645/jmde.v3i4.82 DOI: https://doi.org/10.56645/jmde.v3i4.82
Coryn, C. L. S. (2007). Evaluation of researchers and their research: Toward making the implicit explicit. Unpublished doctoral dissertation, Western Michigan University, Kalamazoo.
Coryn, C. L. S., & Hattie, J. A. (2006). The transdisciplinary model of evaluation. Journal of MultiDisciplinary Evaluation, 3(4), 107-114. https://doi.org/10.56645/jmde.v3i4.81 DOI: https://doi.org/10.56645/jmde.v3i4.81
Coryn, C. L. S., Hattie, J. A., Scriven, M., & Hartmann, D. J. (2007). Models and mechanisms for evaluating government-funded research: An international comparison. American Journal of Evaluation, 28(4), 437-457. https://doi.org/10.1177/1098214007308290 DOI: https://doi.org/10.1177/1098214007308290
Coryn, C. L. S., & Scriven, M. (2008). Reforming the evaluation of research. New Directions for Evaluation: San Francisco, CA: Jossey-Bass. https://doi.org/10.1002/ev.256 DOI: https://doi.org/10.1002/ev.256
Cousins, J. B., & Leithwood, K. A. (1986). Current empirical research on evaluation utilization. Review of Educational Research, 56(3), 331-364. https://doi.org/10.3102/00346543056003331 DOI: https://doi.org/10.3102/00346543056003331
Einstein, A. (1905a). Über einen die Erzeugung and Verwandlung des Lichtes betreffenden heuristischen Gesichtspunkt. Annalen der Physik, 17(2), 132-148. https://doi.org/10.1002/andp.19053220607 DOI: https://doi.org/10.1002/andp.19053220607
Einstein, A. (1905b). Eine neue Bestimmung der Moleküldimensionen. Unpublished doctoral dissertation, Zürich Universität, Zürich.
Einstein, A. (1905c). Über die von der molekülarkinetischen Theorie der Wärme geforderte Bewegung von in ruhenden Flüssigkeiten suspendierten Teilchen. Annalen der Physik, 17(2), 549-560. https://doi.org/10.1002/andp.19053220806 DOI: https://doi.org/10.1002/andp.19053220806
Einstein, A. (1905d). Zur Electrodynamik bewegter Körper. Annalen der Physik, 17(2), 891-921. https://doi.org/10.1002/andp.19053221004 DOI: https://doi.org/10.1002/andp.19053221004
Einstein, A. (1905e). Ist die Trägheit eines Körpers von seinem Energieinhalt abhängig? Annalen der Physik, 18(2), 639-641. https://doi.org/10.1002/andp.19053231314 DOI: https://doi.org/10.1002/andp.19053231314
Foltz, F. A. (2000). The ups and downs of peer review: Making funding choices for science. Bulletin for Science, Technology & Society, 20(6), 427-440. https://doi.org/10.1177/027046760002000601 DOI: https://doi.org/10.1177/027046760002000601
Garcia, R., & Calantone, R. (2002). A critical look at technological innovation typology and innovativeness terminology: A literature review. The Journal of Product Innovation Management, 19(2), 110-132. https://doi.org/10.1111/1540-5885.1920110 DOI: https://doi.org/10.1111/1540-5885.1920110
Guetzkow J., Lamont M., & Mallard G. (2004). What is originality in the humanities and the social sciences? American Sociological Review, 69(2), 90-212. https://doi.org/10.1177/000312240406900203 DOI: https://doi.org/10.1177/000312240406900203
Hackett, E. J. (1997). Peer review in science and science policy. In M. S. Frankel & J. Cave (Eds.), Evaluating science and scientists: An East-West dialogue on research evaluation in post-Communist Europe (pp. 51-60). Budapest, Hungary: Central European University Press. https://doi.org/10.7829/j.ctv280b88f.11 DOI: https://doi.org/10.7829/j.ctv280b88f.11
Julnes, G., & Rog, D. L. (Eds.). (2007). Informing federal policies on evaluation methodology: Building the evidence base for method choice in government sponsored evaluation. New Directions for Evaluation, 113. San Francisco, CA: Jossey-Bass.
Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., & Rennie, D. (1998). Does masking author identity improve peer review quality? A randomized controlled trial. JAMA, 280(3), 240-242. https://doi.org/10.1001/jama.280.3.240 DOI: https://doi.org/10.1001/jama.280.3.240
Langfeldt, L. (2002). Decision-making in expert panels evaluating research: Constraints, processes and bias (Doctoral dissertation, University of Oslo, Norway). ISBN 82-7218-465-6.
Mainguy, G., Motamedi, M. R., & Mietchen, D. (2005). Peer review-The newcomer's perspective. PLoS Biology, 3(9), 1534-1535. https://doi.org/10.1371/journal.pbio.0030326 DOI: https://doi.org/10.1371/journal.pbio.0030326
Mason, G., Blanton, S., McDonald, K., Neal, J., Tanyu, M., Taylor-Ritzler, T., & Reeves, E. (2004). Evaluation 2004: Overall conference survey. Retrieved on May 10, 2007 from http://www.eval.org/training/evalhistory. asp.
McCook, A. (2006). Is peer review broken? The Scientist, 20(2), 26.
McEneaney, J. E. (2001). Electronic submission and review of NRC conference proposals: Re-engineering social literacies for online environments. Paper presented at the meeting of the National Reading Conference. December 7, 2001. San Antonio, TX. Retrieved on June 10, 2007 from http://personalwebs.oakland.edu/~mceneane/nrc/conf01/NRCeProposals.doc
Niederhauser, D.S., Wetzel, K., & Lindstrom, D. L. (2004). From manuscript to article: Publishing educational technology research. Contemporary Issues in Technology and Teacher Education,4(2), 89 -136.
OECD (1987). Evaluation of research: A selection of current practices. Paris, France: Organisation for Economic Co-Operation and Development.
Patton, M. Q. (1997). Utilization-focused evaluation: The new century text (3rd ed.). Thousand Oaks, CA: Sage.
Patton, M. Q., Grimes, P. S, Guthrie, K. M, Brennan, N. J., French, B. D, & Blyth, D. A. (1977). In search of impact: An analysis of the utilization of federal health evaluation research. In C. H. Weiss (Ed.), Using social research in public policy making (pp. 141-164). Lexington, MA: D.C. Heath.
Preskill, H., & Caracelli, V. (1997). Current and developing conceptions of use: Evaluation use TIG survey results. American Journal of Evaluation,18(1), 209-225. https://doi.org/10.1177/109821409701800122 DOI: https://doi.org/10.1177/109821409701800122
Preskill, H., Zuckerman, B., & Matthews, B. (2003). An exploratory study of process use: Findings and implications for future research. American Journal of Evaluation, 24(4),423-442. https://doi.org/10.1177/109821400302400402 DOI: https://doi.org/10.1177/109821400302400402
Pyenson, L., & Sheets-Pyenson, S. (1999). Servants of nature: A history of scientific institutions, enterprises, and sensibilities. New York, NY: W. W. Norton & Company.
Rockwell, S. K., Dickey, E. C., & Jasa, P. J. (1990). The personal factor in evaluation use: A case study of a steering committee's use of a conservation tillage survey. Evaluation and Program Planning, 13(4), 389-394. https://doi.org/10.1016/0149-7189(90)90024-Q DOI: https://doi.org/10.1016/0149-7189(90)90024-Q
Ross, J. S., Gross, C. P., Desai, M. M., Hong, Y., Grant, A. O., Daniels, S. R., Hachinski, V. C., Gibbons, R. J., Gardner, T. J., & Krumholz, H. M. (2006). Effects of blinded peer review on abstract acceptance. JAMA,295(14), 1675-1680. https://doi.org/10.1001/jama.295.14.1675 DOI: https://doi.org/10.1001/jama.295.14.1675
Schröter, D. C. (2007). Learning from AEA TIG proposal review standards. Think tank at the annual conference of the American Evaluation Association, Baltimore, MD. Scriven, M. (1991). The evaluation thesaurus (4thed.). Newbury Park, CA: Sage.
Scriven, M. (1993). Hard-won lessons in program evaluation. New Directions for Program Evaluation, 58. San Francisco, CA: Jossey-Bass. https://doi.org/10.1002/ev.1647 DOI: https://doi.org/10.1002/ev.1647
Scriven, M. (1996). Types of evaluation and types of evaluator. Evaluation Practice, 17(2), 151-161. https://doi.org/10.1177/109821409601700207 DOI: https://doi.org/10.1016/S0886-1633(96)90020-3
Scriven, M. (2003). Evaluation in the new millennium: The transdisciplinary vision. In S. I. Donaldson & M. Scriven (Eds.), Evaluating social programs and problems: Visions for the millennium (pp. 19-42). Mahwah, NJ: Lawrence Erlbaum Associates.
Shadish, W. R., & Epstein, R. (1987). Patterns of program evaluation practice among members of the Evaluation Research Society and Evaluation Network. Evaluation Review, 11(5), 555-590. https://doi.org/10.1177/0193841X8701100501 DOI: https://doi.org/10.1177/0193841X8701100501
Smith, R. (1999). Opening up BMJ peer review: A beginning that should lead to complete transparency. BMJ, 318, 4-5. https://doi.org/10.1136/bmj.318.7175.4 DOI: https://doi.org/10.1136/bmj.318.7175.4
Swindler, S., Hughes, G., Briggs, C., Yamazaki, K., Ohse, D., Pinero, S., & Sagrestano, L. (2002). American Evaluation Association annual conference evaluation: Evaluation 2002 final report. Retrieved on May 10, 2007 from http://www.eval.org/training/evalhistory.asp
Turney, J. (1990). End of the peer show? New Scientist, 22, 38-42.
Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000). Open peer review: a randomized controlled trial. The British Journal of Psychiatry, 176, 47-51. https://doi.org/10.1192/bjp.176.1.47 DOI: https://doi.org/10.1192/bjp.176.1.47
Williams, J. E. (1989). A numerically developed taxonomy of evaluation theory and practice. Evaluation Review, 13(1), 18-31. https://doi.org/10.1177/0193841X8901300102 DOI: https://doi.org/10.1177/0193841X8901300102