Integrating the Funder-Fundee Relationship into a Model of Evaluation Utilization
Main Article Content
Abstract
Background: Evaluation is a tool that can promote accountability and enhance organizational improvement. For these reasons funding entities from government to foundations are increasingly relying on program evaluation as a key instrument to determine effectiveness and hold recipient organizations accountable. What has ensued is an environment of increasing evaluation efforts, where findings often weigh heavily on future funding decisions.
Purpose: The purpose of this article is to examine how the nature of the funder-fundee interorganizational relationship influences evaluation utilization by the fundee organization. Within this context, attention is paid to the role the evaluator plays in the evaluation process and the skills needed to promote utilization.
Setting: A statewide multi-tier public/private initiative addressing early childhood education.
Intervention: Not applicable.
Research Design: A model is designed to test the influence of the nature of the funder-fundee interorganizational relationship on factors associated with utilization, and ultimately on use itself.
Data Collection and Analysis: A survey was administered online and data analysis was conducted using principal components analysis (PCA) and structural equation modeling (SEM).
Findings: The results suggest that the funder-fundee relationship plays a critical role in the evaluation process, and consequently influences utilization. However, the findings indicate that the relationship among the mediating factors is more complex than originally hypothesized.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Copyright and Permissions
Authors retain full copyright for articles published in JMDE. JMDE publishes under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY - NC 4.0). Users are allowed to copy, distribute, and transmit the work in any medium or format for noncommercial purposes, provided that the original authors and source are credited accurately and appropriately. Only the original authors may distribute the article for commercial or compensatory purposes. To view a copy of this license, visit creativecommons.org
References
Alkin, M. C. (1985). A guide for evaluation decision makers. Beverly Hills, CA: Sage.
Alkin, M. C. (2003). Introduction. In T. Kellaghan & D. L. Stufflebeam (Eds.), International handbook of educational evaluation (pp. 189-195). Norwalk, MA: Kluwer Academic. https://doi.org/10.1007/978-94-010-0309-4_13 DOI: https://doi.org/10.1007/978-94-010-0309-4_13
Alkin, M. C., Daillak, R. H., & White, P. (1979). Using evaluations: Does evaluation make a difference? Beverly Hills, CA: Sage.
Alkin, M. C., & Taut, S. M. (2003). Unbundling evaluation use. Studies in Educational Evaluation, 29, 1-12. https://doi.org/10.1016/S0191-491X(03)90001-0 DOI: https://doi.org/10.1016/S0191-491X(03)90001-0
Bailey, D., & Koney K. M. (2000). Strategic alliances among health and human services organizations. Thousand Oaks, CA: Sage. https://doi.org/10.4135/9781483328546 DOI: https://doi.org/10.4135/9781483328546
Baron, R. M., & Kenny D. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51(6), 1173-1182. https://doi.org/10.1037//0022-3514.51.6.1173 DOI: https://doi.org/10.1037//0022-3514.51.6.1173
Bentler, P. M. (1980). Multivariate analysis with latent variables: Causal modeling. Annual Review of Psychology, 31, 419-456. https://doi.org/10.1146/annurev.ps.31.020180.002223 DOI: https://doi.org/10.1146/annurev.ps.31.020180.002223
Campbell, D. (2002). Outcomes assessment and the paradox of nonprofit accountability. Nonprofit Management and Leadership, 12(3), 243-259. https://doi.org/10.1002/nml.12303 DOI: https://doi.org/10.1002/nml.12303
Carman, J. G., & Fredericks, K. A. (2008). Nonprofits and evaluation: Empirical Evidence from the Field. In J. G. Carman & K. A. Fredericks (Eds.), Nonprofits and Evaluation: New Directions for Program Evaluation (Vol. 119, pp. 51-72). San Francisco: Jossey-Bass. https://doi.org/10.1002/ev.268 DOI: https://doi.org/10.1002/ev.268
Child, J., and Faulkner, D. (1998). Strategies of cooperation: Managing alliances, networks, and joint ventures. New York, NY: Oxford University Press. https://doi.org/10.1093/oso/9780198774846.001.0001 DOI: https://doi.org/10.1093/oso/9780198774846.001.0001
Cohen, J., and Cohen, P. R. (1983). Applied multiple regression/correlation analysis for the behavioral sciences (2nd ed.). Hillsdale, N.J.: Lawrence Erlbaum.
Cousins, J. B., and Leithwood, K. A. (1986). Current empirical research on evaluation utilization. Review of Educational Research, 56(3), 331-364. https://doi.org/10.3102/00346543056003331 DOI: https://doi.org/10.3102/00346543056003331
Easterling, D. (2000). Using outcome evaluation to guide grant making: Theory, reality, and possibilities. Nonprofit and Voluntary Sector Quarterly, 29(1), 482-486. https://doi.org/10.1177/0899764000293009 DOI: https://doi.org/10.1177/0899764000293009
Fine, A. H., Thayer, C. E., & Coghlan, A. (1998). Program Evaluation Practice in the Nonprofit Sector: A study funded by The Aspen Institute Nonprofit Sector Research Fund and the Robert Wood Johnson Foundation. Washington, D.C.: Innovation Network, Inc.
Fleischer, D. N., & Christie, C. A. (2009). Evaluation use: Results from a survey of U.S. American Evaluation Association members. American Journal of Evaluation, 30(2), 158-175. https://doi.org/10.1177/1098214008331009 DOI: https://doi.org/10.1177/1098214008331009
Fredericks, K. A., Carman, J. G., & Birkland, T. A. (2002). Program evaluation in a challenging authorizing environment: Intergovernmental and interorganziational factors. New Directions for Evaluation, 95, 5-22. https://doi.org/10.1002/ev.55 DOI: https://doi.org/10.1002/ev.55
Frederickson, G. H. & Smith, K. B. (2003). The public administration theory primer. Boulder, CO: Westview Press.
Frey, B. B., Lohmeier, J. H., Lee, S. W., and Tollefson, N. (2006). Measuring collaboration among grant partners. American Journal of Evaluation, 27(3), 383-392. https://doi.org/10.1177/1098214006290356 DOI: https://doi.org/10.1177/1098214006290356
Gajda, R. (2004). Utilizing collaboration theory to evaluate strategic alliances. American Journal of Evaluation, 25(1), 65-77. https://doi.org/10.1177/109821400402500105 DOI: https://doi.org/10.1177/109821400402500105
Galaskiewicz, J. (1985). Interorganizational relations. Annual Review of Sociology, 11, 281-304 https://doi.org/10.1146/annurev.soc.11.1.281 DOI: https://doi.org/10.1146/annurev.soc.11.1.281
Greene, J. G. (1988). Stakeholder participation and utilization in program evaluation. Evaluation Review, 12(2), 91-116. https://doi.org/10.1177/0193841X8801200201 DOI: https://doi.org/10.1177/0193841X8801200201
Gronbjerg, K. A. (1993). Understanding nonprofit funding. San Francisco: Jossey-Bass.
Henry, G. T. (2000). Why not use? In V. Caracelli and H. Preskill (Eds.), The expanding scope of evaluation use. New Directions for Evaluation (Vol. 88, pp. 85-98). San-Francisco, CA: Jossey-Bass. https://doi.org/10.1002/ev.1193 DOI: https://doi.org/10.1002/ev.1193
Henry, G., & Mark, M. (2003). Beyond use: Understanding evaluation's influence on attitudes and actions. American Journal of Evaluation, 24(3), 293-314. https://doi.org/10.1177/109821400302400302 DOI: https://doi.org/10.1177/109821400302400302
Hoefer, R. (2000). Accountability in action? Program evaluation in nonprofit human service agencies. Nonprofit Management and Leadership, 11(2), 167-177. https://doi.org/10.1002/nml.11203 DOI: https://doi.org/10.1002/nml.11203
Hofstetter, C. H., & Alkin, M. C (2003). Evaluation use revisited. In T. Kellaghan and D. L. Stufflebeam (Eds.), International handbook of educational evaluation (pp. 97-222). Norwalk, MA: Kluwer Academic. https://doi.org/10.1007/978-94-010-0309-4_14 DOI: https://doi.org/10.1007/978-94-010-0309-4_14
Johnson, R. B. (1998). Toward a theoretical model of evaluation utilization. Evaluation and Program Planning, 21, 93-110. https://doi.org/10.1016/S0149-7189(97)00048-7 DOI: https://doi.org/10.1016/S0149-7189(97)00048-7
King, J. A., Stevahn, L., Ghere, G., & Minnema, J. (2001). Toward a taxonomy of essential evaluator competencies. American Journal of Evaluation, 22(2), 229-247. https://doi.org/10.1177/109821400102200206 DOI: https://doi.org/10.1177/109821400102200206
Kirkhart, K. E. (2000). Reconceptualizing evaluation use: An integrated theory of influence. In V. Caracelli and H. Preskill (Eds.), The expanding scope of evaluation use. New Directions for Evaluation (Vol. 88, pp. 5-24). San Francisco, CA: Jossey-Bass. https://doi.org/10.1002/ev.1188 DOI: https://doi.org/10.1002/ev.1188
Leviton, L. C., & Hughes, E. F. X. (1981). Research on the utilization of evaluations: A review and synthesis. Evaluation Review, 5(4), 525-548. https://doi.org/10.1177/0193841X8100500405 DOI: https://doi.org/10.1177/0193841X8100500405
Mark, M. M., Henry, G. T., & Julnes, G. (2000). Evaluation: An integrated framework for understanding, guiding, and improving public and nonprofit policies and programs. San Francisco, CA, Jossey-Bass Inc.
Milward, H. B., & Provan, K. G. (2000). Governing the hallow state. Journal of Public Administration and Research Theory, 10, 359-379. https://doi.org/10.1093/oxfordjournals.jpart.a024273 DOI: https://doi.org/10.1093/oxfordjournals.jpart.a024273
Patton, M. Q. (1978). Utilization-focused evaluation. Beverly Hills, CA: Sage. Patton, M. Q., Grimes, P. S., Guthrie, K.
M., Brennan, B. D. F., & Blyth, D. A. (1977). In search of impact: An analysis of the utilization of federal health evaluation research. In C. H. Weiss (Ed.), Using social research in public policy making (pp. 141-164). Lexington, MA: D.C. Heath and Company.
Preskill, H., & Caracelli, V. J. (1997). Current and developing conceptions of use: Evaluation use topical interest group survey results. Evaluation Practice, 18(3), 209-225. https://doi.org/10.1016/S0886-1633(97)90028-3 DOI: https://doi.org/10.1016/S0886-1633(97)90028-3
Preskill, H. and Torres, R. T. (2000). The learning dimension of evaluation use. In V. Caracelli and H. Preskill (Eds.), The expanding scope of evaluation use. New Directions for Evaluation (Vol. 88, pp. 25-38). San Francisco, CA: Jossey-Bass. https://doi.org/10.1002/ev.1189 DOI: https://doi.org/10.1002/ev.1189
Reitan, T. C. (1998). Theories of interorganizational relations in the human services. Social Service Review, 9, 285-309. https://doi.org/10.1086/515760 DOI: https://doi.org/10.1086/515760
Scriven, M. (1967). The methodology of evaluation. In R. E. Stake (Ed.), American Education Research Association monograph series on curriculum evaluation (Vol. 1, pp. 39- 83). Chicago: Rand McNally.
Shula, L. M., and Cousins, J. B. (1997). Evaluation use: Theory, research, and practice since 1986. Evaluation Practice, 18(3), 195-208. https://doi.org/10.1016/S0886-1633(97)90027-1 DOI: https://doi.org/10.1016/S0886-1633(97)90027-1
Spector, P. E. (2006). Method variance in organizational research. Organizational Research Methods, 9(2), 221-232. https://doi.org/10.1177/1094428105284955 DOI: https://doi.org/10.1177/1094428105284955
Stevahn, L., King, J. A., Ghere, G., & Minnema, J. (2005). Establishing essential competencies for program evaluators. American Journal of Evaluation, 26(1), 43-59. https://doi.org/10.1177/1098214004273180 DOI: https://doi.org/10.1177/1098214004273180
Ullman, J. B. (2001). Structural equation modeling. In B. G. Tabachnick and L. S. Fidell (Eds.), Using multivariate statistics (4th ed.) (pp. 653-771). Boston, MA: Allyn and Bacon.
Van de Ven, A. H., & Ferry, D. L. (1980). Measuring and assessing organizations. New York, NY: Wiley and Sons.
Weiss, C. H. (1972). Utilization of evaluation: Toward comparative study. In C. H. Weiss (Ed.), Evaluating action programs; Readings in social action and education (pp. 318-326). Boston, MA: Allyn and Bacon.
Weiss, C. H. (1979). The many meanings of research utilization. Public Administration Review, 39(5), 426- 431. https://doi.org/10.2307/3109916 DOI: https://doi.org/10.2307/3109916
Weiss, C. H. (1998). Have we learned anything new about the use of evaluation? American Journal of Evaluation, 19(1), 21-33. https://doi.org/10.1016/S1098-2140(99)80178-7 DOI: https://doi.org/10.1016/S1098-2140(99)80178-7